Nov 03 11:50:18.227682 ip-10-0-139-140 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:50:18.605039 ip-10-0-139-140 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:18.605039 ip-10-0-139-140 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:50:18.605039 ip-10-0-139-140 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:18.605039 ip-10-0-139-140 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:50:18.610807 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610722    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.610807 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610808    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610813    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610817    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610820    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610823    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610825    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610828    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610831    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610834    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610837    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610843    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610845    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610848    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610852    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610855    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610859    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610865    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610868    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610871    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.610891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610874    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610877    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610880    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610883    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610888    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610891    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610894    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610897    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610900    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610903    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610906    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610909    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610912    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610915    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610917    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610920    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610925    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610928    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610930    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610933    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.611329 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610935    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610938    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610940    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610943    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610946    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610948    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610951    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610954    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610957    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610962    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610964    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610967    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610970    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610972    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610976    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610979    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610982    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610984    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610987    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.611812 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610991    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610993    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.610998    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611001    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611004    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611006    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611009    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611011    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611014    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611017    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611019    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611022    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611024    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611027    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611030    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611035    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611038    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611041    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611043    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611046    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.612259 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611049    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611052    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611055    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611058    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611061    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611064    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611066    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611072    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611075    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611078    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611081    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611083    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611086    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611089    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611091    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611094    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611097    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611101    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611103    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.612737 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611109    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611111    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611114    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611116    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611119    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611122    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611124    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.611127    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612440    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612449    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612452    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612456    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612460    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612466    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612469    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612473    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612475    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612478    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612481    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.613180 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612484    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612487    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612490    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612493    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612496    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612498    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612504    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612506    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612509    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612512    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612514    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612517    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612519    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612522    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612525    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612527    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612530    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612533    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612535    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612541    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.613682 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612544    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612547    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612550    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612552    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612555    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612558    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612561    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612564    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612567    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612570    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612573    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612578    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612580    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612583    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612586    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612588    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612591    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612593    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612596    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612599    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.614200 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612601    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612604    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612607    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612609    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612614    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612617    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612620    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612623    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612625    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612628    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612630    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612633    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612636    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612639    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612641    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612644    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612647    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612652    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612655    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612658    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.614782 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612663    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612668    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612672    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612675    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612678    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612681    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612684    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612687    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612689    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612695    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612698    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612700    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612703    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612706    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612708    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612711    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612713    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612716    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612719    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.615258 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612721    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612729    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612733    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612736    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612739    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612742    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612745    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612748    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612751    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612753    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612756    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612759    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612762    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612764    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612769    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.612772    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613927    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613934    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613937    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.615764 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613939    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613942    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613945    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613948    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613950    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613954    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613958    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613961    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613966    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613969    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613972    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613974    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613978    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613980    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613983    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613986    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613989    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613992    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613994    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.616210 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.613997    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614000    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614006    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614010    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614014    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614018    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614021    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614023    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614026    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614029    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614032    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614035    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614037    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614040    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614045    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614048    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614050    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614053    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614055    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.616674 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614058    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614061    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614063    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614066    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614068    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614071    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614073    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614076    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614081    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614084    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614087    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614090    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614093    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614095    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614098    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614100    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614103    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614106    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614109    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614112    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.617134 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614115    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614120    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614123    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614126    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614130    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614132    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614135    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614138    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614140    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614143    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614145    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614148    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614150    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614155    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614158    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614160    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614163    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614166    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614168    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614171    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.617630 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614173    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614176    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614179    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614181    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614184    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614189    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614192    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614195    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614197    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614200    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614203    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614206    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614208    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614211    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614214    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614217    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614219    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614223    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614228    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614230    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.618106 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614233    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614236    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614240    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614242    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.614245    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615498    2573 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615517    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615525    2573 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615530    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615535    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615538    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615543    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615547    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615550    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615553    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615557    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615560    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615563    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615566    2573 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615569    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615572    2573 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615575    2573 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615578    2573 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615587    2573 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:50:18.618587 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615590    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615593    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615596    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615600    2573 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615604    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615607    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615611    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615614    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615617    2573 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615620    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615623    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615626    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615629    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615634    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615638    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615641    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615644    2573 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615647    2573 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615650    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615655    2573 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615658    2573 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615661    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615664    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615668    2573 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:50:18.619144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615677    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615680    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615683    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615686    2573 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615689    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615692    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615695    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615698    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615701    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615704    2573 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615707    2573 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615711    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615714    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615717    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615721    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615724    2573 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615727    2573 flags.go:64] FLAG: --help="false"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615730    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615733    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615736    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615739    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615744    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615748    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:50:18.619773 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615751    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615755    2573 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615758    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615761    2573 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615764    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615767    2573 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615770    2573 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615773    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615776    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615780    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615782    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615785    2573 flags.go:64] FLAG: --lock-file=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615788    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615791    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615794    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615800    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615803    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615806    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615810    2573 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615812    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615816    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615819    2573 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615821    2573 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615826    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615829    2573 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:50:18.620312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615834    2573 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615837    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615840    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615849    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615853    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615856    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615859    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615862    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615872    2573 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615875    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615878    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615881    2573 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615884    2573 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615886    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615890    2573 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615893    2573 flags.go:64] FLAG: --port="10250"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615896    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615898    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-011170f36cb618e40"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615902    2573 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615905    2573 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615908    2573 flags.go:64] FLAG: --register-node="true"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615911    2573 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615915    2573 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615918    2573 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615921    2573 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615924    2573 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:50:18.620907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615927    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615930    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615933    2573 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615936    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615939    2573 flags.go:64] FLAG: --runonce="false"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615943    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615946    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615948    2573 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615951    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615954    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615966    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615969    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615972    2573 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615975    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615978    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615982    2573 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615985    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615988    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615991    2573 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.615993    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616000    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616003    2573 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616006    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616010    2573 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616013    2573 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:50:18.621544 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616016    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616019    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616022    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616025    2573 flags.go:64] FLAG: --v="2"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616029    2573 flags.go:64] FLAG: --version="false"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616033    2573 flags.go:64] FLAG: --vmodule=""
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616037    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616041    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616186    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616190    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616194    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616197    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616200    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616202    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616205    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616208    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616210    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616214    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616225    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616230    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.622214 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616235    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616238    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616241    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616244    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616247    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616249    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616252    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616255    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616257    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616260    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616262    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616265    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616268    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616270    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616273    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616275    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616278    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616280    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616283    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616285    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.622724 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616288    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616291    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616293    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616297    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616300    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616302    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616305    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616307    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616310    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616314    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616316    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616320    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616323    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616326    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616329    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616331    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616334    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616336    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616339    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.623198 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616341    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616344    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616346    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616349    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616352    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616354    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616356    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616359    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616361    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616364    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616366    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616369    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616371    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616388    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616392    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616395    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616399    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616401    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616404    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.623664 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616407    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616409    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616412    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616416    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616419    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616421    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616424    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616427    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616430    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616432    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616435    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616437    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616440    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616442    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616445    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616447    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616450    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616453    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616455    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616458    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.624108 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616461    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616463    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616466    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616469    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616471    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616474    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616476    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616479    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616481    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616485    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616487    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616490    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616493    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616495    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616498    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616501    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.624637 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.616530    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616571    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616575    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616579    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616585    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616587    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616590    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616592    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616595    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616598    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616601    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616603    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616606    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616609    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616611    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616614    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616616    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616619    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616622    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.625020 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616624    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616627    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616629    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616632    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616635    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616639    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616642    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616644    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616647    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616649    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616652    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616654    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616658    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616661    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616663    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616666    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616669    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616671    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616674    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.625510 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616676    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616679    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616682    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616685    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616687    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616690    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616694    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616699    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616703    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616706    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616709    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616711    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616714    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616716    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616719    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616721    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616724    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616727    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616731    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.625956 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616734    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616736    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616739    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616741    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616743    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616746    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616750    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616753    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616755    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616758    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616760    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616763    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616766    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616769    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616771    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616774    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616776    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616779    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616781    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.626502 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616784    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616786    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616789    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616792    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616794    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616797    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616799    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616802    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616804    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616807    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616810    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616812    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616817    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616819    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616822    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616824    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616826    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616829    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616831    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.626949 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616835    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616838    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616840    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616843    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616845    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616848    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616852    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616854    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616858    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616860    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.616863    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.623624    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.623640    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.623656    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.623661    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623717    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623722    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623726    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623729    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623732    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.627407 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623735    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623738    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623741    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623743    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623746    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623749    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623752    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623754    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623757    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623760    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623762    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623765    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623768    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623772    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623777    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623780    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623783    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623786    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623789    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.627870 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623792    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623794    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623797    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623800    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623803    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623810    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623813    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623816    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623819    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623821    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623825    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623828    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623830    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623833    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623835    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623838    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623841    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623844    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623846    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.628317 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623849    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623852    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623854    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623857    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623859    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623861    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623864    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623867    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623869    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623872    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623874    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623877    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623880    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623882    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623885    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623888    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623890    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623893    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.628786 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623897    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623901    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623905    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623908    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623911    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623913    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623916    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623918    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623921    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623923    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623926    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623928    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623933    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623936    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623938    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623941    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623943    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623946    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623949    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.629206 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623951    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623954    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623956    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623959    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623961    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623964    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623966    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623969    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623972    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623974    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623977    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623980    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623983    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623986    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623989    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623992    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623994    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.623997    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624000    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624002    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.629728 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624005    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624007    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624010    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624012    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624014    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.624041    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624179    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624183    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624186    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624189    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624192    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624194    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624197    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624199    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624202    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624205    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.630195 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624207    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624210    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624212    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624215    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624217    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624220    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624223    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624226    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624229    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624232    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624234    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624238    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624240    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624243    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624245    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624248    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624250    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624253    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624255    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624258    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.630592 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624260    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624263    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624265    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624268    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624270    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624273    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624275    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624278    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624281    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624283    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624285    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624288    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624290    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624293    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624295    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624298    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624302    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624306    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624309    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624312    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.631052 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624315    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624318    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624321    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624323    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624326    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624329    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624331    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624334    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624336    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624339    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624342    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624344    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624347    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624349    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624352    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624354    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624357    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624359    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624362    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.631617 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624364    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624368    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624371    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624386    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624389    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624391    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624395    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624397    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624400    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624402    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624405    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624408    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624410    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624413    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624416    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624418    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624421    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624423    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624426    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.632056 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624429    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624432    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624434    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624437    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624439    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624442    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624445    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624447    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624449    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624452    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624454    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624457    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624459    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624462    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624465    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624467    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624470    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.632514 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624472    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.624498    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624535    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624539    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624542    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624545    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624548    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624551    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624553    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624556    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624559    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624561    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624564    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624567    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624569    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624572    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:50:18.633082 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624575    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624577    2573 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624580    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624582    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624585    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624587    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624589    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624592    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624594    2573 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624597    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624601    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624605    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624608    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624611    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624613    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624616    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624619    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624621    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624624    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:50:18.633583 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624626    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624629    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624631    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624633    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624636    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624639    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624642    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624645    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624648    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624651    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624653    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624656    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624658    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624662    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624664    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624668    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624671    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624674    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624676    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:50:18.634071 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624679    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624682    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624685    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624687    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624690    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624692    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624695    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624697    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624700    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624702    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624705    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624707    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624710    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624712    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624715    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624718    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624720    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624722    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624725    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:50:18.634529 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624728    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624730    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624733    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624735    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624738    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624740    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624743    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624745    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624748    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624751    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624753    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624756    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624758    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624761    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624763    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624766    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624768    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624771    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624773    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:50:18.635057 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624775    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624778    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624780    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624782    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624785    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624787    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624790    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624792    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624795    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624797    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624801    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624803    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624805    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624808    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624810    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:18.624813    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.624863    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.628516    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.629510    2573 server.go:1014] "Starting client certificate rotation"
Nov 03 11:50:18.635554 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.629607    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:18.636004 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.629633    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:50:18.636004 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.631518    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:18.636004 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.632651    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:50:18.639008 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.638991    2573 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:50:18.642169 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.642156    2573 log.go:25] "Validated CRI v1 image API"
Nov 03 11:50:18.643032 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.643009    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:50:18.648470 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.648448    2573 fs.go:135] Filesystem UUIDs: map[665cefae-866c-4ad0-abbb-d0d7bf324e05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7de24a3a-beb9-4f3b-976b-7ee4807d3db1:/dev/nvme0n1p4]
Nov 03 11:50:18.648555 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.648470    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 11:50:18.653631 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.653525    2573 manager.go:217] Machine: {Timestamp:2026-11-03 11:50:18.652097429 +0000 UTC m=+0.314523929 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3150659 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cadd1676146352c4357b8ec6cca42 SystemUUID:ec2cadd1-6761-4635-2c43-57b8ec6cca42 BootID:cf702fac-b92b-4c0e-8cd3-6242471a95f6 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:dc:b6:f2:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:dc:b6:f2:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:77:e7:f1:73:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:50:18.653631 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.653630    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:50:18.653787 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.653732    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:50:18.656528 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.656498    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:50:18.656656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.656530    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-140.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:50:18.657236 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.657226    2573 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:50:18.657269 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.657237    2573 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:50:18.657269 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.657265    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:18.657321 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.657279    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:50:18.657971 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.657960    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:50:18.661979 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.661967    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:50:18.662098 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662089    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:50:18.662783 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662773    2573 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:50:18.662814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662786    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:50:18.662814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662797    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:50:18.662814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662806    2573 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:50:18.662814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.662814    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:50:18.663125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.663112    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:18.665241 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.665222    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:50:18.670046 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670032    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:50:18.670782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670770    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:50:18.670834 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670800    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:50:18.670947 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670937    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:50:18.671001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670952    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:50:18.671001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670959    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:50:18.671001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670964    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:50:18.671001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670970    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:50:18.671001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.670976    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:50:18.671563 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671554    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:50:18.671593 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671564    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:50:18.671593 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671571    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:50:18.671593 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671578    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:50:18.671593 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671594    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:50:18.671789 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.671603    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:50:18.672147 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.672137    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:50:18.672147 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.672147    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:50:18.675770 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.675757    2573 server.go:1263] "Started kubelet"
Nov 03 11:50:18.675911 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.675840    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:50:18.675966 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.675929    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:50:18.676009 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.675961    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:50:18.676549 ip-10-0-139-140 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:50:18.677037 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.677020    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:50:18.677999 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.677984    2573 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:50:18.683247 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.683170    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 11:50:18.685609 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.685588    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:18.686235 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686213    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:50:18.686362 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.686338    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:18.686914 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686894    2573 factory.go:55] Registering systemd factory
Nov 03 11:50:18.686914 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686916    2573 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:50:18.687055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686898    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:50:18.687055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686899    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:50:18.687055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.686945    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:50:18.687055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687013    2573 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:50:18.687055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687021    2573 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:50:18.687346 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687328    2573 factory.go:153] Registering CRI-O factory
Nov 03 11:50:18.687346 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687347    2573 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:50:18.687483 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687402    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:50:18.687483 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687415    2573 factory.go:103] Registering Raw factory
Nov 03 11:50:18.687483 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.687425    2573 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:50:18.688296 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.688277    2573 manager.go:319] Starting recovery of all containers
Nov 03 11:50:18.696823 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.696805    2573 manager.go:324] Recovery completed
Nov 03 11:50:18.702301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.702269    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.704156 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704130    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.704222 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704162    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.704222 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704176    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.704874 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704858    2573 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:50:18.704874 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704869    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.705001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704874    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:50:18.705001 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.704894    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:50:18.705598 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.705583    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.705665 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.705614    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.705665 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.705628    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.707080 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.707067    2573 policy_none.go:50] "Start"
Nov 03 11:50:18.707080 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.707083    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:50:18.707178 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.707096    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:50:18.709880 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.709864    2573 policy_none.go:44] "Start"
Nov 03 11:50:18.732009 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.731979    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:50:18.734778 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.734762    2573 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:50:18.735317 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.735300    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:50:18.735354 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735320    2573 server.go:87] "Starting device plugin registration server"
Nov 03 11:50:18.735601 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735590    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:50:18.735642 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735604    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:50:18.735888 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735727    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:50:18.735888 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735818    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:50:18.735888 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.735825    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:50:18.736324 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.736307    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:50:18.736395 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.736346    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:18.810216 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.810178    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-140.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 11:50:18.810216 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.810192    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-140.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 11:50:18.812199 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.810167    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-140.us-west-2.compute.internal.18e486aa8d6f1bd4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:18.675731412 +0000 UTC m=+0.338157912,LastTimestamp:2026-11-03 11:50:18.675731412 +0000 UTC m=+0.338157912,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-140.us-west-2.compute.internal,}"
Nov 03 11:50:18.831227 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.831200    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:50:18.832553 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.832530    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:50:18.832637 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.832557    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:50:18.832637 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.832577    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:50:18.832637 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.832620    2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 11:50:18.836109 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.836093    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.836851 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.836835    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.836933 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.836871    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.836933 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.836887    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.836933 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.836918    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.869507 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.869291    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 11:50:18.869507 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.869293    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 11:50:18.869507 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.869331    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-140.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 11:50:18.869507 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.869293    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-140.us-west-2.compute.internal.18e486aa8f20b148  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-140.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:18.70414676 +0000 UTC m=+0.366573258,LastTimestamp:2026-11-03 11:50:18.70414676 +0000 UTC m=+0.366573258,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-140.us-west-2.compute.internal,}"
Nov 03 11:50:18.933217 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.933172    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"]
Nov 03 11:50:18.933357 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.933246    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.934188 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.934168    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.934322 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.934204    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.934322 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.934218    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.936582 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.936566    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.936723 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.936709    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.936765 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.936740    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.937307 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937284    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.937367 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937316    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.937367 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937325    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.937367 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937294    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.937498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937402    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.937498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.937416    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.939443 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.939429    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.939504 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.939453    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:18.940309 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.940286    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:18.940399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.940318    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:18.940399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.940334    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:18.952977 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.952959    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:18.953069 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.953034    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7v7wm"
Nov 03 11:50:18.955174 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.955078    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-140.us-west-2.compute.internal.18e486aa8f21084b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-140.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:18.704169035 +0000 UTC m=+0.366595534,LastTimestamp:2026-11-03 11:50:18.704169035 +0000 UTC m=+0.366595534,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-140.us-west-2.compute.internal,}"
Nov 03 11:50:18.957780 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.957762    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.963075 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:18.963058    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.988465 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.988443    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.988556 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.988473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:18.988556 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:18.988498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.013465 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.013441    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:19.014004 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.013988    2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-139-140.us-west-2.compute.internal\" not found" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089476 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089449    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089476 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089494    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089528    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089545    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.089624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.089561    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.105825 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.105804    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-140.us-west-2.compute.internal\" not found" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.215125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.215053    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:19.216030 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.216012    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:19.216149 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.216046    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:19.216149 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.216060    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:19.216149 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.216092    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.260653 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.260632    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.267210 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.266691    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.268769 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.268752    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:19.268842 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.268777    2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-140.us-west-2.compute.internal\": node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.376937 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.376905    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.477480 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.477372    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.577946 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.577916    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.629089 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.629055    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 11:50:19.629559 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.629251    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 11:50:19.629559 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.629228    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9e0eeeddb0cc409da088ac8135c41d1-80c4a411ee0df651.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-140.us-west-2.compute.internal.18e486aa8f21084b\": read tcp 10.0.139.140:53504->3.139.159.102:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-140.us-west-2.compute.internal.18e486aa8f21084b  default   5231 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-140.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-140.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-140.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 11:50:18 +0000 UTC,LastTimestamp:2026-11-03 11:50:18.934211034 +0000 UTC m=+0.596637535,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-140.us-west-2.compute.internal,}"
Nov 03 11:50:19.678712 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.678669    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.780930 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.780838    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.882039 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.882007    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:19.927285 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.927252    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:50:19.928469 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.928448    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:50:19.928585 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.928487    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:50:19.928585 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.928504    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:50:19.958411 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.958352    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:19 +0000 UTC" deadline="2028-06-25 04:55:38.762728951 +0000 UTC"
Nov 03 11:50:19.958411 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:19.958402    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14393h5m18.804331012s"
Nov 03 11:50:19.983364 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:19.983335    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:20.085265 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:20.085183    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:50:20.100264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.100245    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:20.100429 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.100376    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:20.186475 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.186447    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:20.194504 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.194486    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:20.263046 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.263023    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:20.265557 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.265541    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:50:20.332562 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.332531    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:50:20.455866 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:20.455829    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ff2cd276a94f8c36394a4ab321034f.slice/crio-61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a WatchSource:0}: Error finding container 61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a: Status 404 returned error can't find the container with id 61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a
Nov 03 11:50:20.463091 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.463072    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:50:20.466196 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:20.466175    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fada2c7443a601ca789370ef919cac.slice/crio-96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0 WatchSource:0}: Error finding container 96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0: Status 404 returned error can't find the container with id 96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0
Nov 03 11:50:20.665516 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.665478    2573 apiserver.go:52] "Watching apiserver"
Nov 03 11:50:20.687266 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.687234    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 11:50:20.781553 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.781470    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:50:20.781698 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.781586    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:50:20.781881 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.781864    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-l4fvp","openshift-ovn-kubernetes/ovnkube-node-b7h7z","openshift-image-registry/node-ca-6htsx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-7jh9h","openshift-network-diagnostics/network-check-target-58msl","kube-system/global-pull-secret-syncer-8tx9f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp","openshift-dns/node-resolver-dpmg7","openshift-multus/multus-4gqzk","kube-system/konnectivity-agent-785td","kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m2svl","openshift-multus/network-metrics-daemon-r84qx"]
Nov 03 11:50:20.785718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.785695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.787968 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.787949    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.790206 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.790188    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:20.792602 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.792574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:20.792713 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:20.792668    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:20.794922 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.794892    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.797306 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.797287    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:20.797403 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:20.797346    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:20.799706 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.799689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:20.801565 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801537    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.801660 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801580    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.801660 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.801759 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.801797 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801748    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.801855 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801837    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.801911 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801863    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.801911 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801886    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.801992 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801919    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802169 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.801996    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802351 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvxr5\" (UniqueName: \"kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802405 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.802483 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802445    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.802548 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802548 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802548 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.802693 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802569    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:20.802693 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.802807 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.802699    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.802807 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:20.802646    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:20.803157 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803113    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.803264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.803414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.803414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803365    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803650 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzds\" (UniqueName: \"kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.803650 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.803650 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803506    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.803650 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.803530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.805600 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.805576    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.808029 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.808012    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:20.810550 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.810533    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:20.812972 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.812953    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.836906 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.836867    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"84ff2cd276a94f8c36394a4ab321034f","Type":"ContainerStarted","Data":"61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a"}
Nov 03 11:50:20.837946 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.837915    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerStarted","Data":"96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0"}
Nov 03 11:50:20.844844 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.844663    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.844844 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.844672    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:50:20.844844 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.844701    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.844844 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.844722    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:50:20.844844 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.844801    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:50:20.845655 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.845638    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.845879 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.845865    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:50:20.845976 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.845962    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.847523 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.847509    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:50:20.852020 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.852006    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.852283 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.852247    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:50:20.889297 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.889266    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:50:20.896176 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.896150    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:50:20.897291 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897272    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.897486 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897467    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:50:20.897853 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897833    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:50:20.897853 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897842    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.898010 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897932    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.898010 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.897943    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:50:20.898010 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898008    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:50:20.898164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898147    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.898219 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898166    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:50:20.898279 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898264    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:50:20.898328 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898309    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.898370 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898317    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.898370 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898330    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:50:20.898370 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898356    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:50:20.898515 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898461    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.898515 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898483    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:50:20.898515 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.898504    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:50:20.900018 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900000    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:50:20.900018 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900014    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:50:20.900185 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900075    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 11:50:20.900185 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900113    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:50:20.900185 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900170    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-phc2q"
Nov 03 11:50:20.900306 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900207    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:50:20.900458 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900437    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:50:20.900528 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900466    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:50:20.900676 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.900662    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:50:20.904570 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904554    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.904620 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.904620 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904598    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:20.904620 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.904762 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904633    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.904762 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904735    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.904867 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904777    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.904867 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904818    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.904867 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904854    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.904993 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.904993 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.904993 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.904993 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl2j8\" (UniqueName: \"kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.904984    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905050    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905094    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905174    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905211    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.905402 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905394    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905533    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905633    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.905718 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905672    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905719    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905861    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905850    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905887    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905971    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.905999    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:20.906078 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906067    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906080    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906123    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906220    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906238    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906264    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906476    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906635    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzds\" (UniqueName: \"kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906633    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.906656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906658    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906749    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906809    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906832    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906370    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906939    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.906975    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907079    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907124    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907150    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907169    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqhm\" (UniqueName: \"kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907281    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907301 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907325    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907343    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907359    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcb75\" (UniqueName: \"kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907419    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxr5\" (UniqueName: \"kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907449    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907480    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907523    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907542    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907555    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907608    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907659    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907677    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907710    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907759    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tddxh\" (UniqueName: \"kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.907965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907838    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907902    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2srs\" (UniqueName: \"kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907944    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.907989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908040    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtdh\" (UniqueName: \"kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908091    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908175    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpztv\" (UniqueName: \"kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908215    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:20.908482 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.908290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:20.909807 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.909791    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.909857 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.909845    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:20.970838 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.970810    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxr5\" (UniqueName: \"kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:20.970975 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:20.970822    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzds\" (UniqueName: \"kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:21.009332 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.009492 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:21.009492 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.009617 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009437    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009936    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2j8\" (UniqueName: \"kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.009943    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010186    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.010408 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010318    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.010782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010542    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.010782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.010782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.010782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.010782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010839    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010893    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010914    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010892    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010954    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011002 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011000    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011106    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011179    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011256    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011331    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011353 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011347    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011446    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011502    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011543    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011564    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011563    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.010842    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011639    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011670    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011679    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011733    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011785    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.011900 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011904    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.011966    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012010    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012056    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012108    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqhm\" (UniqueName: \"kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcb75\" (UniqueName: \"kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012287    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012394    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddxh\" (UniqueName: \"kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012526    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2srs\" (UniqueName: \"kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:21.012712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtdh\" (UniqueName: \"kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012629    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012654    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012682    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.012839    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpztv\" (UniqueName: \"kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.012880    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013029    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013099    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013278    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.013521 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013476    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.013562    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:21.513502025 +0000 UTC m=+3.175928532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.013617    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013666    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013720    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.013951    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:21.513931073 +0000 UTC m=+3.176357579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:21.014308 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.013615    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.014659 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014367    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.014659 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014403    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:21.014659 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014478    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.014659 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014552    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.014659 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014625    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.014889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.014658    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.016543 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.016524    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.072462 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.072367    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:21.072462 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.072419    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:21.072462 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.072440    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:21.072685 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.072547    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:21.572523339 +0000 UTC m=+3.234949853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:21.074776 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.074753    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2j8\" (UniqueName: \"kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.075168 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.075151    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqhm\" (UniqueName: \"kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.081273 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.081245    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2srs\" (UniqueName: \"kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:21.081445 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.081427    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcb75\" (UniqueName: \"kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.081498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.081484    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpztv\" (UniqueName: \"kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.081892 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.081859    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtdh\" (UniqueName: \"kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.082023 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.082006    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddxh\" (UniqueName: \"kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.099144 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.099123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4gqzk"
Nov 03 11:50:21.104810 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.104795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:50:21.109864 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.109843    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddab969bc_1102_4323_816b_eb8944cc4d82.slice/crio-e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42 WatchSource:0}: Error finding container e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42: Status 404 returned error can't find the container with id e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42
Nov 03 11:50:21.113757 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.113740    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:50:21.114692 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.114668    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488fc93c_6c34_49ae_83b9_9e6f930ad2b7.slice/crio-c466385652c51cbe2952df49c6bbb0949731d7f63367ac34eb7025e7b9a80955 WatchSource:0}: Error finding container c466385652c51cbe2952df49c6bbb0949731d7f63367ac34eb7025e7b9a80955: Status 404 returned error can't find the container with id c466385652c51cbe2952df49c6bbb0949731d7f63367ac34eb7025e7b9a80955
Nov 03 11:50:21.122830 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.122807    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0c3e91_a0ba_4045_a210_240fbc41be3c.slice/crio-71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea WatchSource:0}: Error finding container 71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea: Status 404 returned error can't find the container with id 71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea
Nov 03 11:50:21.131070 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.131052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:21.133307 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.133284    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:50:21.137877 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.137043    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:50:21.145703 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.145678    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00433242_908e_469a_8d10_39330e427e5a.slice/crio-e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc WatchSource:0}: Error finding container e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc: Status 404 returned error can't find the container with id e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc
Nov 03 11:50:21.147568 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.147544    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b52073_13fa_4418_b8fd_f9866952b917.slice/crio-f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759 WatchSource:0}: Error finding container f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759: Status 404 returned error can't find the container with id f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759
Nov 03 11:50:21.148409 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.148373    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd84fe1e_ce28_4634_b84c_fa50c5dc2be2.slice/crio-fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b WatchSource:0}: Error finding container fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b: Status 404 returned error can't find the container with id fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b
Nov 03 11:50:21.149254 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.149236    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:50:21.158959 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.158936    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4616fa40_654c_46fd_9f47_aeada7820c79.slice/crio-8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72 WatchSource:0}: Error finding container 8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72: Status 404 returned error can't find the container with id 8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72
Nov 03 11:50:21.160985 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.160965    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:50:21.163496 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.163479    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:21.171067 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.171046    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e95d56_77bd_4a2d_b839_326a87300313.slice/crio-8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9 WatchSource:0}: Error finding container 8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9: Status 404 returned error can't find the container with id 8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9
Nov 03 11:50:21.173047 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:50:21.173025    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8d5a65_1ac4_4581_bcc1_0dbcf11d2a72.slice/crio-35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5 WatchSource:0}: Error finding container 35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5: Status 404 returned error can't find the container with id 35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5
Nov 03 11:50:21.516065 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.515967    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:21.516065 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.516021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:21.516300 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.516188    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:21.516300 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.516251    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.516231184 +0000 UTC m=+4.178657690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:21.516454 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.516332    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:21.516454 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.516367    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.516355559 +0000 UTC m=+4.178782067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:21.616956 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.616910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:21.617195 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.617139    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:21.617195 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.617158    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:21.617195 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.617174    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:21.617357 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:21.617229    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:22.617212231 +0000 UTC m=+4.279638734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:21.841672 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.841582    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5"}
Nov 03 11:50:21.843141 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.843106    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp" event={"ID":"c0e95d56-77bd-4a2d-b839-326a87300313","Type":"ContainerStarted","Data":"8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9"}
Nov 03 11:50:21.844746 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.844696    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4fvp" event={"ID":"4616fa40-654c-46fd-9f47-aeada7820c79","Type":"ContainerStarted","Data":"8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72"}
Nov 03 11:50:21.846281 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.846257    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpmg7" event={"ID":"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2","Type":"ContainerStarted","Data":"fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b"}
Nov 03 11:50:21.847864 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.847833    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerStarted","Data":"f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759"}
Nov 03 11:50:21.849248 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.849202    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-785td" event={"ID":"00433242-908e-469a-8d10-39330e427e5a","Type":"ContainerStarted","Data":"e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc"}
Nov 03 11:50:21.850666 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.850640    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6htsx" event={"ID":"0a0c3e91-a0ba-4045-a210-240fbc41be3c","Type":"ContainerStarted","Data":"71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea"}
Nov 03 11:50:21.852128 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.852080    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m2svl" event={"ID":"488fc93c-6c34-49ae-83b9-9e6f930ad2b7","Type":"ContainerStarted","Data":"c466385652c51cbe2952df49c6bbb0949731d7f63367ac34eb7025e7b9a80955"}
Nov 03 11:50:21.854505 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.854484    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gqzk" event={"ID":"dab969bc-1102-4323-816b-eb8944cc4d82","Type":"ContainerStarted","Data":"e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42"}
Nov 03 11:50:21.904051 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.904009    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:21 +0000 UTC" deadline="2028-06-18 22:30:55.275789296 +0000 UTC"
Nov 03 11:50:21.904051 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:21.904048    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14242h40m33.371746379s"
Nov 03 11:50:22.525141 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.525093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:22.525312 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.525160    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:22.525312 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.525304    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:22.525420 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.525371    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:24.525352471 +0000 UTC m=+6.187778973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:22.525563 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.525458    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:22.525563 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.525498    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:24.525484381 +0000 UTC m=+6.187910885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:22.625615 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.625571    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:22.625812 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.625750    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:22.625812 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.625771    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:22.625812 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.625785    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:22.625975 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.625851    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:24.625832751 +0000 UTC m=+6.288259261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:22.834743 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.834644    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:22.834908 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.834751    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:22.834908 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.834816    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:22.834908 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.834862    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:22.837108 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.834610    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:22.837307 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:22.837284    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:22.904876 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.904802    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:21 +0000 UTC" deadline="2028-07-29 14:13:50.8574943 +0000 UTC"
Nov 03 11:50:22.904876 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:22.904868    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15218h23m27.952630415s"
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.544791    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.544842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.545020    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.545087    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:28.545065723 +0000 UTC m=+10.207492215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.545184    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:24.545304 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.545224    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:28.545211736 +0000 UTC m=+10.207638228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:24.647061 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.646748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:24.647061 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.646918    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:24.647061 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.646942    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:24.647061 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.646956    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:24.647061 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.647012    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:28.646993472 +0000 UTC m=+10.309420000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:24.833423 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.833186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:24.833567 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.833541    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:24.833619 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.833302    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:24.833666 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.833648    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:24.833666 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.833254    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:24.833765 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:24.833744    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:24.880223 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.880181    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"84ff2cd276a94f8c36394a4ab321034f","Type":"ContainerStarted","Data":"afb9eda708279ee84ea94d0c08b02b8ce57ec5117c95e75caaff4cac7a35befa"}
Nov 03 11:50:24.883569 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:24.883295    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerStarted","Data":"2c3e2c05753628746b8f432f622c07abb25412436c2f571fc4226c358bc7e718"}
Nov 03 11:50:25.131341 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:25.131268    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal" podStartSLOduration=4.131246673 podStartE2EDuration="4.131246673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:25.005377945 +0000 UTC m=+6.667804468" watchObservedRunningTime="2026-11-03 11:50:25.131246673 +0000 UTC m=+6.793673229"
Nov 03 11:50:25.894967 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:25.894443    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d73e6682745ed6f0797339fba701ddfd784ced035a7056674a0880ad3824682d" exitCode=0
Nov 03 11:50:25.894967 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:25.894537    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"d73e6682745ed6f0797339fba701ddfd784ced035a7056674a0880ad3824682d"}
Nov 03 11:50:25.910111 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:25.910073    2573 generic.go:356] "Generic (PLEG): container finished" podID="71fada2c7443a601ca789370ef919cac" containerID="2c3e2c05753628746b8f432f622c07abb25412436c2f571fc4226c358bc7e718" exitCode=0
Nov 03 11:50:25.910969 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:25.910922    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerDied","Data":"2c3e2c05753628746b8f432f622c07abb25412436c2f571fc4226c358bc7e718"}
Nov 03 11:50:26.835375 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:26.835343    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:26.835574 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:26.835500    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:26.835688 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:26.835580    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:26.835688 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:26.835665    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:26.835867 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:26.835725    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:26.835867 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:26.835810    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:26.915946 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:26.915908    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-785td" event={"ID":"00433242-908e-469a-8d10-39330e427e5a","Type":"ContainerStarted","Data":"bef54944ad37370d8a1d6bb6858bb6d224d22851cbb25581ab7cc5fa1df63f0c"}
Nov 03 11:50:26.918921 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:26.918895    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerStarted","Data":"5c99352e39372fae136f24732b55bbcf2b89f2517b133597f7a80ec8a32c3a0f"}
Nov 03 11:50:27.042325 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:27.042240    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-785td" podStartSLOduration=2.339258218 podStartE2EDuration="7.042223605s" totalImagesPullingTime="4.702965387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:27.04186692 +0000 UTC m=+8.704293434" watchObservedRunningTime="2026-11-03 11:50:27.042223605 +0000 UTC m=+8.704650116"
Nov 03 11:50:27.168059 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:27.167175    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" podStartSLOduration=6.167155109 podStartE2EDuration="6.167155109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:27.166519235 +0000 UTC m=+8.828945747" watchObservedRunningTime="2026-11-03 11:50:27.167155109 +0000 UTC m=+8.829581624"
Nov 03 11:50:27.925271 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:27.925147    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp" event={"ID":"c0e95d56-77bd-4a2d-b839-326a87300313","Type":"ContainerStarted","Data":"c90bea23c6bc862bc1ffa6bac7690b63ce4bcbbf5a056e9319efa9be590954e2"}
Nov 03 11:50:27.930029 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:27.929546    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6htsx" event={"ID":"0a0c3e91-a0ba-4045-a210-240fbc41be3c","Type":"ContainerStarted","Data":"2f121d75719e3fdca7b410fb70cf6b881763932d884ab33dd6606e5bc51e25fb"}
Nov 03 11:50:28.054480 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.054400    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6htsx" podStartSLOduration=1.7394515990000001 podStartE2EDuration="8.054378374s" totalImagesPullingTime="6.314926775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:28.053677781 +0000 UTC m=+9.716104294" watchObservedRunningTime="2026-11-03 11:50:28.054378374 +0000 UTC m=+9.716804886"
Nov 03 11:50:28.581334 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.581289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:28.581558 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.581352    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:28.581558 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.581533    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:28.581687 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.581604    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:36.581584073 +0000 UTC m=+18.244010577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:28.581687 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.581674    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:28.581806 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.581709    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:36.581697107 +0000 UTC m=+18.244123611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:28.682265 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.682186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:28.682452 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.682379    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:28.682452 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.682425    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:28.682452 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.682441    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:28.682643 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.682517    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:36.682496423 +0000 UTC m=+18.344922928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:28.836651 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.836574    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:28.836822 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.836731    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:28.836895 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.836825    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:28.836955 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.836920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:28.838137 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.838110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:28.838239 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:28.838200    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:28.935878 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:28.935839    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dpmg7" event={"ID":"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2","Type":"ContainerStarted","Data":"8649463b1b29dea973cedb5cc6fa0a0e16e9148e46831aafd796b3c66757b6d5"}
Nov 03 11:50:29.062597 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.062536    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-dpmg7" podStartSLOduration=1.5111765560000001 podStartE2EDuration="9.06252029s" totalImagesPullingTime="7.551343734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:29.061896807 +0000 UTC m=+10.724323321" watchObservedRunningTime="2026-11-03 11:50:29.06252029 +0000 UTC m=+10.724946802"
Nov 03 11:50:29.424442 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.424405    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:50:29.745322 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.745156    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:50:29.424427789Z","UUID":"683d078e-d8c0-44fe-ac74-63b39c5e4a7b","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:50:29.747437 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.746996    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:50:29.747437 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.747031    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:50:29.952136 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.952102    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp" event={"ID":"c0e95d56-77bd-4a2d-b839-326a87300313","Type":"ContainerStarted","Data":"191142d05d21fa67eb68a135c5735cd6e35a7526396844394909b4c7f4af71bb"}
Nov 03 11:50:29.953574 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:29.953548    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4gqzk" event={"ID":"dab969bc-1102-4323-816b-eb8944cc4d82","Type":"ContainerStarted","Data":"42767f1e62c44ba07e2725e0ee7f2b66279e5959eee84db87a82315fe607e764"}
Nov 03 11:50:30.084664 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.084615    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4gqzk" podStartSLOduration=2.340441752 podStartE2EDuration="10.0846012s" totalImagesPullingTime="7.744159448s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:30.084351444 +0000 UTC m=+11.746777955" watchObservedRunningTime="2026-11-03 11:50:30.0846012 +0000 UTC m=+11.747027712"
Nov 03 11:50:30.287326 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.287063    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:30.833759 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.833728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:30.833917 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:30.833874    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:30.833984 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.833958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:30.834120 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:30.834099    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:30.834200 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.834169    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:30.834250 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:30.834233    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:30.959399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:30.959317    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l4fvp" event={"ID":"4616fa40-654c-46fd-9f47-aeada7820c79","Type":"ContainerStarted","Data":"dc20e1a6088cb7ad1bb6be21fc90364c5891ba3b4f8221898cc4264c20a6fa73"}
Nov 03 11:50:31.074080 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.074018    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l4fvp" podStartSLOduration=2.972944467 podStartE2EDuration="11.074004264s" totalImagesPullingTime="8.101059797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:31.073794175 +0000 UTC m=+12.736220676" watchObservedRunningTime="2026-11-03 11:50:31.074004264 +0000 UTC m=+12.736430772"
Nov 03 11:50:31.207943 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.205405    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:31.207943 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.205938    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:31.962604 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.962569    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp" event={"ID":"c0e95d56-77bd-4a2d-b839-326a87300313","Type":"ContainerStarted","Data":"69a3b537d0fc7b489c9a325db9692c20a44fa9b15886fe8bc3771a689a5833d9"}
Nov 03 11:50:31.964511 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.964478    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d90518b10c6adbbe7c3c8f497542122a546f0617a9e9fb08dea0857b494fe7cb" exitCode=0
Nov 03 11:50:31.964640 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.964568    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"d90518b10c6adbbe7c3c8f497542122a546f0617a9e9fb08dea0857b494fe7cb"}
Nov 03 11:50:31.965582 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:31.965269    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:50:32.080829 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:32.080774    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp" podStartSLOduration=2.42329409 podStartE2EDuration="12.08075839s" totalImagesPullingTime="9.6574643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:32.080482703 +0000 UTC m=+13.742909212" watchObservedRunningTime="2026-11-03 11:50:32.08075839 +0000 UTC m=+13.743184899"
Nov 03 11:50:32.833269 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:32.833228    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:32.833269 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:32.833248    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:32.833495 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:32.833348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:32.833495 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:32.833405    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:32.833563 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:32.833504    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:32.833605 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:32.833580    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:33.970223 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:33.970187    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="29c906b063361d5719b3505fc4548ce796e1bda0de85345256cb633a36dc8ad3" exitCode=0
Nov 03 11:50:33.970810 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:33.970243    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"29c906b063361d5719b3505fc4548ce796e1bda0de85345256cb633a36dc8ad3"}
Nov 03 11:50:34.833168 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:34.833135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:34.833342 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:34.833270    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:34.833342 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:34.833139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:34.833342 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:34.833317    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:34.833482 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:34.833420    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:34.833644 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:34.833560    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:35.979020 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:35.978975    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="65a962bbfcfca4e91e0bc7571ca6f0c06d97e11fec39f365df57f4a2fb2466cc" exitCode=0
Nov 03 11:50:35.979528 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:35.979035    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"65a962bbfcfca4e91e0bc7571ca6f0c06d97e11fec39f365df57f4a2fb2466cc"}
Nov 03 11:50:36.644035 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.643998    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:36.644035 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.644041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:36.644366 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.644127    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:36.644366 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.644133    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:36.644366 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.644204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.644182504 +0000 UTC m=+34.306609020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:36.644366 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.644231    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.644214747 +0000 UTC m=+34.306641236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:36.745014 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.744982    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:36.745207 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.745188    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:36.745253 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.745213    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:36.745253 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.745228    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:36.745325 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.745291    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:50:52.745273302 +0000 UTC m=+34.407699790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:36.833438 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.833403    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:36.833438 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.833444    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:36.833677 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:36.833394    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:36.833677 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.833550    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:36.833791 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.833642    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:36.833858 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:36.833784    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:38.834566 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.834245    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:38.834889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.834295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:38.834889 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:38.834663    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:38.834889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.834323    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:38.834889 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:38.834767    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:38.834889 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:38.834799    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:38.988673 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988582    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"8f1951b58b182ff94511ceb55c3654f4c1c1e98301495c2db9ffd731b919dd79"}
Nov 03 11:50:38.988673 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988665    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"a315d3c6c00178e7a454c853d38a5610eec0fe0794762a95d4638eacb16edcd0"}
Nov 03 11:50:38.988892 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988708    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"c6ef54fc327871bcc717b1a613a3607903858b05fa688c15ff813e1313401ab6"}
Nov 03 11:50:38.988892 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988755    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"bdedd0f29502239550674cf043e3446f3cf322d4cd015bf35f7a77c609189afb"}
Nov 03 11:50:38.988892 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988794    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"5cc89f68a265c2216b0865b00f5d28837b3477d95a35aecf00f5aa97ea3fc101"}
Nov 03 11:50:38.988892 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.988835    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"ccd5bc26a41cb11b44fe99ca57064e2a2de44e614e32e50af40a5939f68c0335"}
Nov 03 11:50:38.991323 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.991296    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d26218639fcb239885b8bc00f6a579b6f935d908a50282482e43d780a4743fd7" exitCode=0
Nov 03 11:50:38.991447 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:38.991341    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"d26218639fcb239885b8bc00f6a579b6f935d908a50282482e43d780a4743fd7"}
Nov 03 11:50:39.853441 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:39.853412    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:39.996133 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:39.996101    2573 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="6ffd97eeda26ae86779a494b029d8751b9961e01634ffd71f124c52e11f120c1" exitCode=0
Nov 03 11:50:39.996321 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:39.996150    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerDied","Data":"6ffd97eeda26ae86779a494b029d8751b9961e01634ffd71f124c52e11f120c1"}
Nov 03 11:50:40.081388 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:40.081353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:40.296253 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:40.296126    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:40.833133 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:40.833106    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:40.833133 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:40.833128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:40.833304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:40.833110    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:40.833304 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:40.833242    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:40.833419 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:40.833311    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:40.833419 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:40.833414    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:41.001211 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:41.001176    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" event={"ID":"47b52073-13fa-4418-b8fd-f9866952b917","Type":"ContainerStarted","Data":"ff35e27311ba9c850fb3f8708ae589e0d95efd47e47e36942c656fe664fd69b6"}
Nov 03 11:50:41.004218 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:41.004194    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"91a564de06f2aaa02243d299314e19b5ae552299667aaec8dfafad2626130953"}
Nov 03 11:50:41.125046 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:41.124949    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7jh9h" podStartSLOduration=7.105964317 podStartE2EDuration="21.124935213s" totalImagesPullingTime="14.018970896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:41.12450908 +0000 UTC m=+22.786935587" watchObservedRunningTime="2026-11-03 11:50:41.124935213 +0000 UTC m=+22.787361720"
Nov 03 11:50:42.833177 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:42.833137    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:42.833177 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:42.833161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:42.833808 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:42.833140    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:42.833808 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:42.833270    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:42.833808 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:42.833368    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:42.833808 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:42.833442    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:43.011672 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:43.011636    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" event={"ID":"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72","Type":"ContainerStarted","Data":"96a7c61fb059f419187436f494beed65a67f00bbb641f96a97b11035fd95d78d"}
Nov 03 11:50:43.011884 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:43.011867    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:43.027337 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:43.027310    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:43.290490 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:43.290393    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z" podStartSLOduration=5.992078003 podStartE2EDuration="23.290366205s" totalImagesPullingTime="17.298288202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:43.238155577 +0000 UTC m=+24.900582085" watchObservedRunningTime="2026-11-03 11:50:43.290366205 +0000 UTC m=+24.952792763"
Nov 03 11:50:44.014040 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.014002    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:44.014861 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.014079    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:44.030594 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.030570    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:50:44.462188 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.462112    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:44.628454 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.628425    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:44.833049 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.832951    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:44.833049 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.832998    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:44.833049 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:44.832962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:44.833269 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:44.833121    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:44.833269 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:44.833209    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:44.833364 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:44.833321    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:45.606827 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.606572    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58msl"]
Nov 03 11:50:45.607298 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.606854    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:45.607298 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:45.606969    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:45.609442 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.609416    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tx9f"]
Nov 03 11:50:45.609547 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.609507    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:45.609617 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:45.609598    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:45.623564 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.623531    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r84qx"]
Nov 03 11:50:45.623748 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:45.623733    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:45.623997 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:45.623938    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:46.833530 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:46.833489    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:46.833911 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:46.833623    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:47.022690 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:47.022652    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m2svl" event={"ID":"488fc93c-6c34-49ae-83b9-9e6f930ad2b7","Type":"ContainerStarted","Data":"311f8b81570998c088ff32f4e94b31f3b4d12fbb3e5fd4fbf4ab3da16ec2ca12"}
Nov 03 11:50:47.140292 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:47.140239    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m2svl" podStartSLOduration=2.165763592 podStartE2EDuration="27.140224523s" totalImagesPullingTime="24.974460931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:50:47.140169144 +0000 UTC m=+28.802595651" watchObservedRunningTime="2026-11-03 11:50:47.140224523 +0000 UTC m=+28.802651031"
Nov 03 11:50:47.833211 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:47.833028    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:47.833399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:47.833108    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:47.833399 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:47.833305    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:47.833399 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:47.833360    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:48.336963 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.336933    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:48.512405 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.512362    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:48.673764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.673728    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:48.833938 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.833884    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:48.834990 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.834960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:48.835599 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:48.835103    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:48.993453 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:48.993342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:49.676415 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:49.676368    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:49.833672 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:49.833588    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:49.833672 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:49.833634    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:49.833873 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:49.833728    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:49.833873 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:49.833843    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:49.847619 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:49.847595    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:50.369125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:50.369097    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:50.833435 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:50.833406    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:50.833888 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:50.833507    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:51.833394 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:51.833357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:51.833555 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:51.833475    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:51.833555 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:51.833527    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:51.833895 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:51.833644    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:52.216840 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.216811    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:52.561851 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.561781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:52.653791 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.653756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:52.653791 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.653799    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:52.654005 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.653902    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:52.654005 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.653902    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:52.654005 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.653960    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs podName:1a5605fa-4135-4de2-a8ab-1d984d5c51be nodeName:}" failed. No retries permitted until 2026-11-03 11:51:24.653939965 +0000 UTC m=+66.316366453 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs") pod "network-metrics-daemon-r84qx" (UID: "1a5605fa-4135-4de2-a8ab-1d984d5c51be") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 11:50:52.654005 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.653973    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret podName:3ece716d-9af9-434f-85bf-9e6e6fb988f3 nodeName:}" failed. No retries permitted until 2026-11-03 11:51:24.653967455 +0000 UTC m=+66.316393941 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret") pod "global-pull-secret-syncer-8tx9f" (UID: "3ece716d-9af9-434f-85bf-9e6e6fb988f3") : object "kube-system"/"original-pull-secret" not registered
Nov 03 11:50:52.738989 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.738960    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:52.754073 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.754046    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:52.754208 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.754199    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 11:50:52.754247 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.754215    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 11:50:52.754247 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.754225    2573 projected.go:196] Error preparing data for projected volume kube-api-access-djdqp for pod openshift-network-diagnostics/network-check-target-58msl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:52.754309 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.754269    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp podName:fd4c7517-246c-40d4-84fc-bcdebf2728bb nodeName:}" failed. No retries permitted until 2026-11-03 11:51:24.754255624 +0000 UTC m=+66.416682124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-djdqp" (UniqueName: "kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp") pod "network-check-target-58msl" (UID: "fd4c7517-246c-40d4-84fc-bcdebf2728bb") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 11:50:52.833494 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:52.833430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:52.833671 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:52.833523    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:53.330565 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:53.330532    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:53.833419 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:53.833390    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:53.833573 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:53.833488    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:53.833573 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:53.833516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:53.833640 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:53.833581    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:54.671519 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:54.671488    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:54.827550 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:54.827520    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:54.833579 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:54.833557    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:54.833692 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:54.833648    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:55.833305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:55.833267    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:55.833712 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:55.833401    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:55.833712 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:55.833434    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:55.833712 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:55.833529    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:55.975731 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:55.975694    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:56.564614 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:56.564575    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:56.767185 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:56.767151    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:56.833321 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:56.833255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:56.833698 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:56.833352    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:56.994177 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:56.994147    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:57.597759 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:57.597723    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:57.833771 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:57.833737    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:57.834225 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:57.833839    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:57.834225 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:57.833902    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:57.834225 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:57.834018    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:50:57.910182 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:57.910149    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:58.182580 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.182504    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:58.391398 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.391353    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:58.762273 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.762249    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dpmg7_fd84fe1e-ce28-4634-b84c-fa50c5dc2be2/dns-node-resolver/0.log"
Nov 03 11:50:58.811018 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.810988    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:58.834366 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.834340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:50:58.834755 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:58.834433    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:50:58.976824 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:58.976794    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:59.130463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:59.130432    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:59.408354 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:59.408278    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:50:59.561722 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:59.561696    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6htsx_0a0c3e91-a0ba-4045-a210-240fbc41be3c/node-ca/0.log"
Nov 03 11:50:59.833118 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:59.833044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:50:59.833264 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:59.833153    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:50:59.833264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:50:59.833209    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:50:59.833341 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:50:59.833295    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:00.192247 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:00.192218    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:00.612058 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:00.611970    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:00.820446 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:00.820413    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:00.833481 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:00.833457    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:00.833596 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:00.833575    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:00.984345 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:00.984297    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:01.142644 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:01.142610    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:01.397959 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:01.397929    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:01.833840 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:01.833760    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:01.833840 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:01.833796    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:01.834010 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:01.833872    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:01.834010 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:01.833998    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:02.197752 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:02.197703    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:02.601760 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:02.601669    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:02.788577 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:02.788542    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:02.832980 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:02.832956    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:02.833101 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:02.833079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:02.995121 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:02.995085    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:03.402560 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:03.402531    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:03.833427 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:03.833331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:03.833427 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:03.833331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:03.833811 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:03.833460    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:03.833811 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:03.833526    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:04.199594 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:04.199558    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:04.596085 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:04.596008    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:04.757751 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:04.757717    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:04.833882 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:04.833853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:04.834253 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:04.833960    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:04.911063 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:04.911034    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.071711 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.071674    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.227265 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.227176    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.392894 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.392865    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.561857 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.561781    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.723782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.723752    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:05.832956 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.832881    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:05.832956 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:05.832910    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:05.833133 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:05.832997    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:05.833133 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:05.833115    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:06.186035 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:06.186005    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:06.393414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:06.393386    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:06.585494 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:06.585400    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:06.791300 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:06.791230    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:06.833106 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:06.833082    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:06.833246 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:06.833198    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:07.020973 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:07.020942    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:07.188277 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:07.188246    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:07.832956 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:07.832925    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:07.832956 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:07.832954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:07.833155 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:07.833022    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:07.833155 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:07.833099    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:08.565391 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:08.565342    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:08.833797 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:08.833724    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:08.833933 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:08.833802    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:09.675306 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:09.675270    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:09.833160 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:09.833130    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:09.833328 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:09.833227    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:09.833328 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:09.833286    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:09.833426 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:09.833399    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:09.851492 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:09.851464    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:10.833838 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:10.833806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:10.834220 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:10.833914    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:11.832836 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:11.832802    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:11.833006 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:11.832910    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r84qx" podUID="1a5605fa-4135-4de2-a8ab-1d984d5c51be"
Nov 03 11:51:11.833006 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:11.832962    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:11.833074 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:11.833047    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8tx9f" podUID="3ece716d-9af9-434f-85bf-9e6e6fb988f3"
Nov 03 11:51:12.034083 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.034048    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:12.277518 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.277470    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:12.441788 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.441755    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:12.605284 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.605207    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:12.833390 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.833332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:12.833551 ip-10-0-139-140 kubenswrapper[2573]: E1103 11:51:12.833455    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-58msl" podUID="fd4c7517-246c-40d4-84fc-bcdebf2728bb"
Nov 03 11:51:12.905631 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:12.905602    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:13.059814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.059779    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:13.254888 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.254817    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:13.613429 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.613342    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeReady"
Nov 03 11:51:13.613563 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.613473    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 11:51:13.781040 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.781003    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jhqfb"]
Nov 03 11:51:13.783059 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.783042    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mw5bn"]
Nov 03 11:51:13.783193 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.783178    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.784879 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.784861    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gk9d8"]
Nov 03 11:51:13.784984 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.784967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.786870 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.786852    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.799418 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.799498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.799498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.799588 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.799635 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.799680 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8zd\" (UniqueName: \"kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.799727 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbtzw\" (UniqueName: \"kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.799771 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.799815 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.799860 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4ks\" (UniqueName: \"kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.799905 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.799860    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.832743 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.832707    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw5bn"]
Nov 03 11:51:13.832891 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.832875    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:13.833821 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.833348    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:13.835410 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.835385    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk9d8"]
Nov 03 11:51:13.836033 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.836015    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhqfb"]
Nov 03 11:51:13.837047 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.837029    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:51:13.837150 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.837072    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:51:13.837245 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.837233    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:51:13.838441 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.838410    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:51:13.838583 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.838554    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:51:13.838652 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.838599    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:51:13.838707 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.838692    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:51:13.883628 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.883601    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:51:13.883628 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.883616    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:51:13.886249 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.886223    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:51:13.886372 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.886356    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:51:13.886435 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.886397    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:51:13.886775 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.886760    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:51:13.888526 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.888512    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:51:13.888588 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.888526    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:51:13.888588 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.888577    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:51:13.900943 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.900920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901054 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.900946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8zd\" (UniqueName: \"kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901054 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.900985    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtzw\" (UniqueName: \"kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.901162 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901162 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901135    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901259 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901166    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4ks\" (UniqueName: \"kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901259 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901189    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901259 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901259 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901247    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901484 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901273    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901484 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901303    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.901582 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901505    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901582 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901566    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901685 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901662    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.901685 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901670    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.901927 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.901902    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.905120 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.905096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:13.905228 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.905205    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.905288 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.905237    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.959414 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.959363    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtzw\" (UniqueName: \"kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:13.959574 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.959456    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4ks\" (UniqueName: \"kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:13.959574 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:13.959472    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8zd\" (UniqueName: \"kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:14.096762 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.096726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:14.101406 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.101353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:14.109290 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.109267    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:14.481729 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.481703    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk9d8"]
Nov 03 11:51:14.522472 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:14.522274    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5025802c_cc81_4a98_8248_8878e9f36e10.slice/crio-db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05 WatchSource:0}: Error finding container db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05: Status 404 returned error can't find the container with id db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05
Nov 03 11:51:14.524510 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.524485    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw5bn"]
Nov 03 11:51:14.525777 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:14.525755    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3aadf1_19c3_41bc_b242_6230cfcac9c3.slice/crio-75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a WatchSource:0}: Error finding container 75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a: Status 404 returned error can't find the container with id 75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a
Nov 03 11:51:14.527351 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.527310    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhqfb"]
Nov 03 11:51:14.833226 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.833150    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:14.887656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.887630    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:51:14.887656 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.887630    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:51:14.887853 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:14.887665    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:51:15.087726 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.087648    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk9d8" event={"ID":"200522d1-e408-41f4-be88-15c3600c7225","Type":"ContainerStarted","Data":"9a04ba3dcf2ad794c83879caa1a4a28a8f9c559d4ada1252ece47b12fef88d15"}
Nov 03 11:51:15.087726 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.087713    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk9d8" event={"ID":"200522d1-e408-41f4-be88-15c3600c7225","Type":"ContainerStarted","Data":"54738eeb368b15c3704971bd789516e986643a08424acc8a7232b5a7fd511c71"}
Nov 03 11:51:15.088778 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.088757    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw5bn" event={"ID":"5025802c-cc81-4a98-8248-8878e9f36e10","Type":"ContainerStarted","Data":"db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05"}
Nov 03 11:51:15.089786 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.089769    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqfb" event={"ID":"ee3aadf1-19c3-41bc-b242-6230cfcac9c3","Type":"ContainerStarted","Data":"75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a"}
Nov 03 11:51:15.222130 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.222097    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:51:15.223815 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.223799    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.279000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.278951    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 11:51:15.279228 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279016    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 11:51:15.279228 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279118    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 11:51:15.279228 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279200    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 11:51:15.279399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279318    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 11:51:15.279399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279341    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 11:51:15.279399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279358    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 11:51:15.279399 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.279389    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sfhxq\""
Nov 03 11:51:15.280219 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.280201    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:51:15.310907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.310881    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.310907 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.310907    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.311066 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.310923    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.311066 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.310945    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.311066 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.311026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.311156 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.311069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.311156 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.311088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412116 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412304 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412590 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412315    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.412723 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.412657    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.413365 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.413343    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.413967 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.413944    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.414534 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.414512    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.415361 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.415337    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.415483 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.415464    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.471486 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.471449    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.533461 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.533185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:15.920673 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:15.920620    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb543b127_aa97_4f98_863e_ca16072e4031.slice/crio-b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51 WatchSource:0}: Error finding container b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51: Status 404 returned error can't find the container with id b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51
Nov 03 11:51:15.922099 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:15.922075    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:51:16.031050 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.030982    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:16.094339 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.094300    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dfb58d96-wn4hk" event={"ID":"b543b127-aa97-4f98-863e-ca16072e4031","Type":"ContainerStarted","Data":"b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51"}
Nov 03 11:51:16.096445 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.096421    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk9d8" event={"ID":"200522d1-e408-41f4-be88-15c3600c7225","Type":"ContainerStarted","Data":"69957fea6c171a2cced4a54cfd8acf4de5283909a51747495100c6c4ff232a1d"}
Nov 03 11:51:16.255711 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.255669    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:16.413393 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.413341    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:16.573300 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.573265    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:16.630264 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.630231    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"]
Nov 03 11:51:16.632742 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.632720    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.688458 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688429    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 11:51:16.688606 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688515    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 11:51:16.688670 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688624    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 11:51:16.688723 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688680    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 11:51:16.688769 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688730    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 11:51:16.688814 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.688805    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-phz8z\""
Nov 03 11:51:16.689546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.689479    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"]
Nov 03 11:51:16.728577 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.728547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.728700 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.728609    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.728745 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.728694    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.728832 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.728812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8wz\" (UniqueName: \"kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.736651 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.736624    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:16.829628 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.829596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.829772 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.829683    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8wz\" (UniqueName: \"kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.829772 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.829744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.829891 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.829792    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.831828 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.830559    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.834608 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.833705    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.834608 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.833974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.891073 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.891011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8wz\" (UniqueName: \"kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:16.946513 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:16.946415    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:17.104244 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:17.104209    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqfb" event={"ID":"ee3aadf1-19c3-41bc-b242-6230cfcac9c3","Type":"ContainerStarted","Data":"7e8867b3cd13074414236c580b5fac3a88eb62a1d91ae61f212a63253ecb9b25"}
Nov 03 11:51:17.104448 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:17.104273    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhqfb" event={"ID":"ee3aadf1-19c3-41bc-b242-6230cfcac9c3","Type":"ContainerStarted","Data":"7d23b744fcd7acdf089b8f2c0403fbc34d4b8f672413e23b0e6bd9172343e61b"}
Nov 03 11:51:17.105258 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:17.105237    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:17.326909 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:17.326039    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-jhqfb" podStartSLOduration=2.193661683 podStartE2EDuration="4.326018471s" totalImagesPullingTime="2.132356788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:17.22222315 +0000 UTC m=+58.884649659" watchObservedRunningTime="2026-11-03 11:51:17.326018471 +0000 UTC m=+58.988444980"
Nov 03 11:51:17.326909 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:17.326639    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"]
Nov 03 11:51:18.108089 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:18.108051    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8" event={"ID":"7aaede73-1b46-4d67-a336-377d44b8c286","Type":"ContainerStarted","Data":"cc001d20f08a1eb68068443f7cd82fddea5f8af1143fc5f8d4da4841a28f4847"}
Nov 03 11:51:19.113195 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:19.113155    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw5bn" event={"ID":"5025802c-cc81-4a98-8248-8878e9f36e10","Type":"ContainerStarted","Data":"a006e553c09df5a4f4cc1e9301bf5b1554bc967eadb9ce3ae6d1705a3c1adb21"}
Nov 03 11:51:19.115619 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:19.115587    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk9d8" event={"ID":"200522d1-e408-41f4-be88-15c3600c7225","Type":"ContainerStarted","Data":"8847128b4a0f02184070d97b687eff38d03e235f9525c950be84e204ae02d968"}
Nov 03 11:51:19.230337 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:19.230272    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mw5bn" podStartSLOduration=2.29704106 podStartE2EDuration="6.230253864s" totalImagesPullingTime="3.933212804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:19.229314063 +0000 UTC m=+60.891740577" watchObservedRunningTime="2026-11-03 11:51:19.230253864 +0000 UTC m=+60.892680373"
Nov 03 11:51:19.349836 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:19.349771    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gk9d8" podStartSLOduration=2.513549113 podStartE2EDuration="6.349753366s" totalImagesPullingTime="3.836204253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 11:51:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:19.34915667 +0000 UTC m=+61.011583183" watchObservedRunningTime="2026-11-03 11:51:19.349753366 +0000 UTC m=+61.012179875"
Nov 03 11:51:20.122514 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:20.122427    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dfb58d96-wn4hk" event={"ID":"b543b127-aa97-4f98-863e-ca16072e4031","Type":"ContainerStarted","Data":"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"}
Nov 03 11:51:20.124164 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:20.124141    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8" event={"ID":"7aaede73-1b46-4d67-a336-377d44b8c286","Type":"ContainerStarted","Data":"a3c405bf239b11c71930674d93d67e075fcdcc9bd3c1bcb2ebb9c4d1d9ed75e4"}
Nov 03 11:51:20.124283 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:20.124179    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8" event={"ID":"7aaede73-1b46-4d67-a336-377d44b8c286","Type":"ContainerStarted","Data":"ca03c13c6c6b9b2fd8356b4e101f8da1b60c757476f45fc22c4e418965983bd0"}
Nov 03 11:51:20.241734 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:20.241679    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-86dfb58d96-wn4hk" podStartSLOduration=1.557793016 podStartE2EDuration="5.241666571s" totalImagesPullingTime="3.683873555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:20.240306601 +0000 UTC m=+61.902733109" watchObservedRunningTime="2026-11-03 11:51:20.241666571 +0000 UTC m=+61.904093079"
Nov 03 11:51:20.358352 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:20.358302    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8" podStartSLOduration=2.290856675 podStartE2EDuration="4.358288391s" totalImagesPullingTime="2.067431716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:20.357854901 +0000 UTC m=+62.020281408" watchObservedRunningTime="2026-11-03 11:51:20.358288391 +0000 UTC m=+62.020714902"
Nov 03 11:51:21.775884 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.775816    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:21.871775 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.871744    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:51:21.874711 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.874695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.937865 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.937831    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:51:21.943187 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.943134    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:21.973592 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973564    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973708 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973598    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973708 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973618    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973708 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973687    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973802 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973802 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973789    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973863 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:21.973895 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:21.973870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.040055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.039979    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 11:51:22.074645 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.074645 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074647    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.074845 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.074845 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074684    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.074845 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.074845 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074715    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075035 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075035 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.074942    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075318 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.075291    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075526 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.075504    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075597 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.075543    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075684 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.075662    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.075684 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.075674    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.077261 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.077241    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.077444 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.077425    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.133180 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.133146    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.185220 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.185186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:22.358471 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.358329    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:22.548369 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.548333    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:51:22.578581 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:22.578551    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c1d3e1_850e_4d74_be2b_d9f1ce17d40a.slice/crio-96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607 WatchSource:0}: Error finding container 96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607: Status 404 returned error can't find the container with id 96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607
Nov 03 11:51:22.607711 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.607651    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:22.768463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.768429    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:22.932290 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.932256    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"]
Nov 03 11:51:22.935292 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.935276    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:22.950749 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.950720    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:22.983070 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.983036    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrq9z\" (UniqueName: \"kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:22.983240 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.983110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:22.983240 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.983188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:22.983240 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.983234    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:22.989927 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.989900    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"]
Nov 03 11:51:22.990886 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.990862    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 11:51:22.991015 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.990900    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 11:51:22.991015 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:22.990964    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jslfh\""
Nov 03 11:51:23.032816 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.032787    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"]
Nov 03 11:51:23.035837 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.035823    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.050470 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.050449    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wrh8c"]
Nov 03 11:51:23.053503 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.053486    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.083971 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.083941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084117 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.083984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.084117 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084008    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084117 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084085    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbmq\" (UniqueName: \"kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084214 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084141    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084246 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084209    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084246 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084236    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084314 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084314 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084281    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpbnk\" (UniqueName: \"kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084403 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrq9z\" (UniqueName: \"kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.084454 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084534 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084588 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084539    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084645 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084645 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.084722 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.084754 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.084754 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.084862 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.084848    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.085448 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.085428    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.086749 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.086724    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.086950 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.086932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.093389 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.093350    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"]
Nov 03 11:51:23.094338 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.094315    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 11:51:23.094455 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.094398    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 11:51:23.094455 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.094441    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 11:51:23.094555 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.094489    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nhpqx\""
Nov 03 11:51:23.125241 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.125212    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:23.133784 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.133755    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db5589b4-9v2nm" event={"ID":"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a","Type":"ContainerStarted","Data":"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"}
Nov 03 11:51:23.133912 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.133796    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db5589b4-9v2nm" event={"ID":"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a","Type":"ContainerStarted","Data":"96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607"}
Nov 03 11:51:23.144360 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.144326    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrq9z\" (UniqueName: \"kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.186222 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.186222 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186222 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186215    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186254    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186463 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186438    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186626    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.186764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbmq\" (UniqueName: \"kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.186764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186691    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.186764 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186873    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbnk\" (UniqueName: \"kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.187000 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.186943    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.187275 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.187122    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.187499 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.187481    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.187603 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.187586    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.188727 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.188707    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.188889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.188872    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.197396 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.197366    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 11:51:23.198092 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.198076    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8zh8\""
Nov 03 11:51:23.198167 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.198093    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 11:51:23.198224 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.198192    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 11:51:23.207623 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.207603    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.209300 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.209285    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.209354 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.209304    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.245573 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.245540    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbnk\" (UniqueName: \"kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.245697 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.245634    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbmq\" (UniqueName: \"kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.250209 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.250190    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:23.351209 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.351177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:23.367710 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.367681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:23.377961 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:23.377923    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36293e5_2420_4631_ad8b_5747f97896f1.slice/crio-1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8 WatchSource:0}: Error finding container 1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8: Status 404 returned error can't find the container with id 1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8
Nov 03 11:51:23.427557 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.427532    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:23.454461 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.454357    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5db5589b4-9v2nm" podStartSLOduration=2.454343613 podStartE2EDuration="2.454343613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:23.45344117 +0000 UTC m=+65.115867679" watchObservedRunningTime="2026-11-03 11:51:23.454343613 +0000 UTC m=+65.116770121"
Nov 03 11:51:23.656969 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:23.656938    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e91a382_0182_42f7_aa04_29c13b922a49.slice/crio-9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92 WatchSource:0}: Error finding container 9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92: Status 404 returned error can't find the container with id 9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92
Nov 03 11:51:23.659112 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.659091    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"]
Nov 03 11:51:23.678553 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.678526    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:23.716266 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:23.716240    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c439b40_37df_4d6b_82ce_004d3275ad8e.slice/crio-171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48 WatchSource:0}: Error finding container 171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48: Status 404 returned error can't find the container with id 171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48
Nov 03 11:51:23.718181 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:23.718158    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"]
Nov 03 11:51:24.139462 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.139429    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh" event={"ID":"5e91a382-0182-42f7-aa04-29c13b922a49","Type":"ContainerStarted","Data":"c26a976bc7e27d05c3c38f078af23f186c7100c836929ee5127c31e2c20cf552"}
Nov 03 11:51:24.139885 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.139504    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh" event={"ID":"5e91a382-0182-42f7-aa04-29c13b922a49","Type":"ContainerStarted","Data":"d63defb387694c1e61f3ef3171e6869ded2a5974a5a836b6b422d0a1cc389f0a"}
Nov 03 11:51:24.139885 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.139537    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh" event={"ID":"5e91a382-0182-42f7-aa04-29c13b922a49","Type":"ContainerStarted","Data":"9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92"}
Nov 03 11:51:24.140560 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.140543    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s" event={"ID":"6c439b40-37df-4d6b-82ce-004d3275ad8e","Type":"ContainerStarted","Data":"171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48"}
Nov 03 11:51:24.141636 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.141617    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrh8c" event={"ID":"a36293e5-2420-4631-ad8b-5747f97896f1","Type":"ContainerStarted","Data":"1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8"}
Nov 03 11:51:24.359641 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.359603    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:24.537624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.537553    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:24.698258 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.698226    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:24.702302 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.702277    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:24.702448 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.702309    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:24.755741 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.755709    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:51:24.755741 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.755714    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:51:24.765662 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.765638    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:24.765799 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.765750    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:24.803257 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.803182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:24.856213 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.856177    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:51:24.865294 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.865266    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:24.917581 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.917554    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:51:24.928069 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:24.928044    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:25.000948 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.000910    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:51:25.003828 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.003801    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:51:25.007916 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.007894    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:25.011392 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.011083    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:25.096642 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.096517    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:51:25.103624 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.103471    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:25.173583 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.173109    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:25.177258 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.177063    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:51:25.184623 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.184600    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.239867    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240031    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240047    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240073    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240149    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240166    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-46zdg\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240188    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240201    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 11:51:25.240305 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.240259    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 11:51:25.291944 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.291909    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 11:51:25.307243 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307421 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307258    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307421 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307421 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307569 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307426    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307569 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307643 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307643 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307635    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307738 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307782 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307750    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307848 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307822    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307966 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.307966 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.307903    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.397400 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.397347    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.408960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409058    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409182    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409237    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409313    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.409691 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.409339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.410444 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.410072    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.410444 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.410189    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.411490 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.411371    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.412050 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.412026    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.413500 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.412873    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8tx9f"]
Nov 03 11:51:25.415120 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.415096    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.415628 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.415604    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.415928 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.415811    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.416259 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.416238    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.416648 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.416619    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.416738 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.416714    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.417262 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.417230    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.418057 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.418035    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.455604 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:25.455566    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a5605fa_4135_4de2_a8ab_1d984d5c51be.slice/crio-19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64 WatchSource:0}: Error finding container 19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64: Status 404 returned error can't find the container with id 19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64
Nov 03 11:51:25.456921 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.456894    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r84qx"]
Nov 03 11:51:25.469391 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.469359    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.504891 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:25.504858    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4c7517_246c_40d4_84fc_bcdebf2728bb.slice/crio-333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61 WatchSource:0}: Error finding container 333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61: Status 404 returned error can't find the container with id 333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61
Nov 03 11:51:25.505097 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.505023    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-58msl"]
Nov 03 11:51:25.514909 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.514734    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:25.526742 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.526715    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7748596f65-48dhl"]
Nov 03 11:51:25.534244 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.532047    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.534244 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.533632    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:25.534244 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.533952    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:25.539990 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.539967    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:25.589785 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.589748    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7748596f65-48dhl"]
Nov 03 11:51:25.590927 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.590902    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 11:51:25.591033 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.590996    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9vqqu7n71jhmb\""
Nov 03 11:51:25.591098 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.591073    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 11:51:25.591145 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.591109    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 11:51:25.591192 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.591166    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 11:51:25.591635 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.591258    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 11:51:25.591635 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.591336    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k6jvk\""
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljvp\" (UniqueName: \"kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611615    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611664    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611698    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611726    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611850    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.612344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.611876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713227 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljvp\" (UniqueName: \"kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713359 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713359 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713300    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713359 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713480    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.713546 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.713513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.715093 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.714269    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.715093 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.715021    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.716648 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.716609    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.716765 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.716708    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.717040 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.717019    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.717580 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.717557    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.718096 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.718073    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.718702 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.718683    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.771949 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.771890    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljvp\" (UniqueName: \"kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.873769 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.873731    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:25.898441 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:25.898415    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c41b69d_ee39_45cb_8a06_66b5029f2952.slice/crio-666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8 WatchSource:0}: Error finding container 666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8: Status 404 returned error can't find the container with id 666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8
Nov 03 11:51:25.899484 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:25.899458    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:51:26.152251 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.152214    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh" event={"ID":"5e91a382-0182-42f7-aa04-29c13b922a49","Type":"ContainerStarted","Data":"a551401dc5f464ff0f92971dd714652c6c387e4db06d566aeb3840ed29b76dab"}
Nov 03 11:51:26.153311 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.153288    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8tx9f" event={"ID":"3ece716d-9af9-434f-85bf-9e6e6fb988f3","Type":"ContainerStarted","Data":"0b65be0eac35cb561a5f27f92c5b29669af9de0808824a20c48e6d52252a0623"}
Nov 03 11:51:26.154529 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.154509    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r84qx" event={"ID":"1a5605fa-4135-4de2-a8ab-1d984d5c51be","Type":"ContainerStarted","Data":"19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64"}
Nov 03 11:51:26.156477 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.156452    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s" event={"ID":"6c439b40-37df-4d6b-82ce-004d3275ad8e","Type":"ContainerStarted","Data":"f8253e0cf5361aab97064dee1899a17d5f096a6056cb45ffeeb189f4b9948692"}
Nov 03 11:51:26.156592 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.156504    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s" event={"ID":"6c439b40-37df-4d6b-82ce-004d3275ad8e","Type":"ContainerStarted","Data":"e26966ae441aca538e2c71a5eb28caf367a661c675c9538f59cc0dd30c78f9fe"}
Nov 03 11:51:26.156592 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.156531    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s" event={"ID":"6c439b40-37df-4d6b-82ce-004d3275ad8e","Type":"ContainerStarted","Data":"f7c44b57d30d07de2d18add54812328894166dadcb34516863438882e3c81345"}
Nov 03 11:51:26.157605 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.157582    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58msl" event={"ID":"fd4c7517-246c-40d4-84fc-bcdebf2728bb","Type":"ContainerStarted","Data":"333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61"}
Nov 03 11:51:26.159014 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.158991    2573 generic.go:356] "Generic (PLEG): container finished" podID="a36293e5-2420-4631-ad8b-5747f97896f1" containerID="1103a83aae7ff99b4eeabf4cc2f3906ed3655e7ce9f8813baf93cb85ca4fb837" exitCode=0
Nov 03 11:51:26.159096 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.159056    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrh8c" event={"ID":"a36293e5-2420-4631-ad8b-5747f97896f1","Type":"ContainerDied","Data":"1103a83aae7ff99b4eeabf4cc2f3906ed3655e7ce9f8813baf93cb85ca4fb837"}
Nov 03 11:51:26.160354 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.160335    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8"}
Nov 03 11:51:26.165348 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.165326    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:26.241504 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:26.241462    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14e9e71_6399_459d_9ade_004b2390424a.slice/crio-aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a WatchSource:0}: Error finding container aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a: Status 404 returned error can't find the container with id aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a
Nov 03 11:51:26.243208 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.243183    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7748596f65-48dhl"]
Nov 03 11:51:26.270911 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.270862    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh" podStartSLOduration=2.554862993 podStartE2EDuration="4.270847723s" totalImagesPullingTime="1.71598473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:26.270074219 +0000 UTC m=+67.932500727" watchObservedRunningTime="2026-11-03 11:51:26.270847723 +0000 UTC m=+67.933274231"
Nov 03 11:51:26.389082 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.388988    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s" podStartSLOduration=2.49081291 podStartE2EDuration="4.388973664s" totalImagesPullingTime="1.898160754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:26.387665598 +0000 UTC m=+68.050092106" watchObservedRunningTime="2026-11-03 11:51:26.388973664 +0000 UTC m=+68.051400172"
Nov 03 11:51:26.829345 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.829310    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:26.985206 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:26.985168    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:27.169043 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.169007    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrh8c" event={"ID":"a36293e5-2420-4631-ad8b-5747f97896f1","Type":"ContainerStarted","Data":"1bab4e04fe417b75916c39437bc13882d4ccd0ea34802b8e98f3b627798ba0f3"}
Nov 03 11:51:27.169288 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.169074    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wrh8c" event={"ID":"a36293e5-2420-4631-ad8b-5747f97896f1","Type":"ContainerStarted","Data":"56f9632af68cfd7d08efe94099a2ea11fec007f7f449d5a94b8566eb7ec71585"}
Nov 03 11:51:27.174498 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.174462    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a"}
Nov 03 11:51:27.185456 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.185424    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:27.288454 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.288369    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wrh8c" podStartSLOduration=1.880169497 podStartE2EDuration="4.288348981s" totalImagesPullingTime="2.336349728s" totalInitContainerRuntime="71.829756ms" isStatefulPod=true podCreationTimestamp="2026-11-03 11:51:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:27.287304996 +0000 UTC m=+68.949731515" watchObservedRunningTime="2026-11-03 11:51:27.288348981 +0000 UTC m=+68.950775491"
Nov 03 11:51:27.333918 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.333827    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:27.616162 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.616082    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:27.806620 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.806490    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:27.907801 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.906431    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-96685b96b-5s9hm"]
Nov 03 11:51:27.911474 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.911446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:27.964403 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.964360    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-96685b96b-5s9hm"]
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.965906    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.966010    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.966100    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x96q5\""
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.966179    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.966290    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a2tbc0bg4loul\""
Nov 03 11:51:27.966502 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.966367    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 11:51:27.973286 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:27.973261    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:28.036774 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.036729    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.036924 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.036781    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.036924 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.036861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.037060 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.036967    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.037060 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.037012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.037060 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.037047    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.037201 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.037080    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cvj\" (UniqueName: \"kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.118327 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.118255    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:28.138465 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138428    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138618 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138481    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138618 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138618 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cvj\" (UniqueName: \"kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138787 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138787 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138652    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.138787 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.138685    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.139265 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.139237    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.139580 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.139558    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.139709 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.139684    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.142103 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.142076    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.142256 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.142234    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.142510 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.142491    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.179889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.179086    2573 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2" exitCode=0
Nov 03 11:51:28.179889 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.179435    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2"}
Nov 03 11:51:28.183345 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.182806    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r84qx" event={"ID":"1a5605fa-4135-4de2-a8ab-1d984d5c51be","Type":"ContainerStarted","Data":"a4b602c9b9f7000290c21c12f8cf47b9db1fce69df804cee89096710577386d1"}
Nov 03 11:51:28.183345 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.182864    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r84qx" event={"ID":"1a5605fa-4135-4de2-a8ab-1d984d5c51be","Type":"ContainerStarted","Data":"1b7477722cf2b95e0bc4b28fd3235e2f1174fda56994ce0616f59854f353c7c3"}
Nov 03 11:51:28.198150 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.198124    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cvj\" (UniqueName: \"kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.249533 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.249505    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:28.703562 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.703504    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r84qx" podStartSLOduration=66.942039144 podStartE2EDuration="1m8.703483996s" totalImagesPullingTime="1.761444852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:28.479456848 +0000 UTC m=+70.141883357" watchObservedRunningTime="2026-11-03 11:51:28.703483996 +0000 UTC m=+70.365910505"
Nov 03 11:51:28.704066 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:28.703727    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-96685b96b-5s9hm"]
Nov 03 11:51:28.704066 ip-10-0-139-140 kubenswrapper[2573]: W1103 11:51:28.703786    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f19698_00a5_4e9e_a7c3_3fd1a8742421.slice/crio-deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1 WatchSource:0}: Error finding container deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1: Status 404 returned error can't find the container with id deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1
Nov 03 11:51:29.189760 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:29.189723    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm" event={"ID":"49f19698-00a5-4e9e-a7c3-3fd1a8742421","Type":"ContainerStarted","Data":"deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1"}
Nov 03 11:51:29.980118 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:29.980057    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:30.136809 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.136775    2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:51:30.141296 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.141264    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:30.165087 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.165054    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:51:30.168462 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.168127    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.201988 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.201951    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"417495f6770aadaf11c75fe3bacfe80b3ded07a2e3b28f36ccce663ad3cb3802"}
Nov 03 11:51:30.202149 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.202044    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"0d6a23d43eff56562d89072141d4a1a5e7422ed2edd056239eb0c91cb24b36cd"}
Nov 03 11:51:30.202149 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.202088    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"778089a93c1d23c57bc2b9ef26b3b26f803f8922eb44ae4ddbad61459687adda"}
Nov 03 11:51:30.243314 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.242697    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:51:30.265098 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265059    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265252 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265124    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265252 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265252 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265165    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265252 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265181    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265252 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265219    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265421 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265259    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.265421 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.265276    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.365911 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.365875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366041 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.365956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366041 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.365980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366138 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366040    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366138 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366104    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366138 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366129    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366233 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.366233 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366180    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.367847 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.366792    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.367847 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.367206    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.367847 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.367330    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.367847 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.367544    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.367847 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.367804    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.382032 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.381848    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.384405 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.382520    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.428655 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.428622    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.502856 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.502603    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:30.918965 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.918944    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:51:30.923662 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:30.923597    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:31.088572 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.088458    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:31.210503 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.210289    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf444864-cm8fl" event={"ID":"3d4550a1-3f78-4685-aa82-2546c2aa7b41","Type":"ContainerStarted","Data":"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"}
Nov 03 11:51:31.210503 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.210340    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf444864-cm8fl" event={"ID":"3d4550a1-3f78-4685-aa82-2546c2aa7b41","Type":"ContainerStarted","Data":"27aec2e03b371cfc540ac49b3e2eb09aa4dacb1b279086a360ed217f3321116d"}
Nov 03 11:51:31.214811 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.214785    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"370fd1712c126af1e792b39aeab16da06736abd52025355c0eed3463369494c8"}
Nov 03 11:51:31.214923 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.214844    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"7739d444ee92ca45a97eeda9a46bc77f3427a95b0249420ca4c30f7822d569cc"}
Nov 03 11:51:31.218984 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.218947    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"}
Nov 03 11:51:31.219125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.219022    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"}
Nov 03 11:51:31.219125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.219065    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"}
Nov 03 11:51:31.219125 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.219104    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"}
Nov 03 11:51:31.219263 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.219142    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"}
Nov 03 11:51:31.331403 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.330896    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-bbf444864-cm8fl" podStartSLOduration=1.330876743 podStartE2EDuration="1.330876743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:31.330095056 +0000 UTC m=+72.992521566" watchObservedRunningTime="2026-11-03 11:51:31.330876743 +0000 UTC m=+72.993303253"
Nov 03 11:51:31.818346 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:31.818312    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:32.174344 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.174313    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:32.185387 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.185336    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:32.226823 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.226787    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-58msl" event={"ID":"fd4c7517-246c-40d4-84fc-bcdebf2728bb","Type":"ContainerStarted","Data":"62e98645eef93a0d554a5d6dc828c542cd447327acd244141dcd7a41347f44ff"}
Nov 03 11:51:32.226986 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.226973    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:32.231067 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.231037    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerStarted","Data":"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"}
Nov 03 11:51:32.235255 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.235214    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" event={"ID":"f14e9e71-6399-459d-9ade-004b2390424a","Type":"ContainerStarted","Data":"6154335e16d2d633fe105d16008c7cdc5768e61b5ebcde3df88430f437ba4805"}
Nov 03 11:51:32.236055 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.236033    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:32.238687 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.238274    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm" event={"ID":"49f19698-00a5-4e9e-a7c3-3fd1a8742421","Type":"ContainerStarted","Data":"5b492ff85e5efcedf4ac1196df5539e221e7cc5a2caa449503215b3de98a1098"}
Nov 03 11:51:32.329540 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.329512    2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:32.345692 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.345636    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-58msl" podStartSLOduration=66.091212288 podStartE2EDuration="1m12.34561629s" totalImagesPullingTime="6.254404002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:32.344540987 +0000 UTC m=+74.006967498" watchObservedRunningTime="2026-11-03 11:51:32.34561629 +0000 UTC m=+74.008042799"
Nov 03 11:51:32.466291 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.466187    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl" podStartSLOduration=2.877529838 podStartE2EDuration="7.466169389s" totalImagesPullingTime="4.588639551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:32.465683519 +0000 UTC m=+74.128110024" watchObservedRunningTime="2026-11-03 11:51:32.466169389 +0000 UTC m=+74.128595897"
Nov 03 11:51:32.591032 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.590977    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.800312111 podStartE2EDuration="7.59095959s" totalImagesPullingTime="4.790647479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:32.589588916 +0000 UTC m=+74.252015425" watchObservedRunningTime="2026-11-03 11:51:32.59095959 +0000 UTC m=+74.253386099"
Nov 03 11:51:32.715555 ip-10-0-139-140 kubenswrapper[2573]: I1103 11:51:32.715489    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm" podStartSLOduration=3.17785107 podStartE2EDuration="5.71546832s" totalImagesPullingTime="2.53761725s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:51:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:51:32.711212294 +0000 UTC m=+74.373638804" watchObservedRunningTime="2026-11-03 11:51:32.71546832 +0000 UTC m=+74.377894828"
Nov 03 11:51:33.479994 ip-10-0-139-140 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 11:51:33.494739 ip-10-0-139-140 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 11:51:33.494977 ip-10-0-139-140 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 11:51:33.495040 ip-10-0-139-140 systemd[1]: kubelet.service: Consumed 4.600s CPU time.
Nov 03 11:51:33.508033 ip-10-0-139-140 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 11:51:33.588098 ip-10-0-139-140 kubenswrapper[5784]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:51:33.588098 ip-10-0-139-140 kubenswrapper[5784]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 11:51:33.588098 ip-10-0-139-140 kubenswrapper[5784]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:51:33.588098 ip-10-0-139-140 kubenswrapper[5784]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 11:51:33.590183 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590098    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.590183 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590182    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.590183 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590187    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590190    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590193    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590196    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590198    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590201    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590204    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590206    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590209    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590212    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590214    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590217    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590219    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590222    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590225    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590227    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590229    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590232    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590234    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590237    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.590280 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590240    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590243    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590245    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590248    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590252    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590260    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590265    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590269    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590272    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590275    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590278    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590281    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590284    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590286    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590290    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590293    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590295    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590298    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.590780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590301    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590303    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590306    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590309    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590312    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590314    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590318    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590320    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590323    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590326    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590328    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590331    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590334    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590336    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590339    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590342    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590345    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590347    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590350    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590352    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.591249 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590355    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590359    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590363    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590366    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590370    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590373    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590389    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590395    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590399    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590404    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590408    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590412    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590417    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590421    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590424    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590427    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590429    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590432    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590435    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590438    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.591739 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590441    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590443    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590446    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590449    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590451    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590454    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590456    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590459    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590461    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590465    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590468    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590470    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590473    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590475    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590478    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590480    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590483    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590486    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590489    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.592192 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590491    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590494    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590496    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590499    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590502    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590504    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.590507    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591268    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591274    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591277    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591280    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591285    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591288    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591293    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591296    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591299    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591303    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591307    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591310    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.592646 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591313    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591316    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591319    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591322    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591325    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591328    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591330    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591333    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591336    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591339    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591342    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591345    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591349    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591351    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591354    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591357    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591359    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591362    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591364    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591367    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.593104 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591369    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591372    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591390    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591392    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591395    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591398    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591401    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591404    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591407    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591409    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591412    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591414    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591417    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591420    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591422    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591425    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591427    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591430    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591433    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591435    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.593589 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591438    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591441    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591444    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591447    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591450    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591453    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591455    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591458    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591460    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591463    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591466    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591468    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591471    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591473    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591475    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591478    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591480    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591483    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591485    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.594068 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591488    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591491    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591494    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591496    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591499    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591501    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591503    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591506    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591509    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591512    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591514    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591517    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591519    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591522    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591524    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591527    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591530    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591532    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591535    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.594527 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591537    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591540    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591543    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591545    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591548    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591550    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591553    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591555    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591558    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591561    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591563    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591566    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591568    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591571    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591573    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.591576    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592062    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592067    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592070    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.594985 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592072    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592076    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592079    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592082    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592084    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592087    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592089    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592092    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592095    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592097    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592100    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592103    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592105    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592108    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592110    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592113    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592118    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592121    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592124    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.595458 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592127    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592131    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592134    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592137    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592140    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592144    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592148    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592151    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592154    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592157    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592159    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592162    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592164    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592167    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592171    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592173    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592176    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592178    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592181    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.595914 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592184    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592186    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592189    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592191    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592194    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592197    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592200    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592203    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592205    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592208    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592212    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592214    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592217    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592220    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592222    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592225    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592227    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592230    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592232    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592235    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.596421 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592237    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592240    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592242    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592245    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592248    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592251    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592253    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592256    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592259    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592262    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592264    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592267    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592270    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592273    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592275    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592278    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592280    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592283    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592285    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.596891 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592288    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592290    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592293    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592297    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592299    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592302    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592304    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592307    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592310    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592312    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592315    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592317    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592320    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592323    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592326    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592328    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592331    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592334    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592336    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592339    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.597343 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592342    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592344    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592347    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592349    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592352    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.592354    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592468    5784 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592478    5784 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592489    5784 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592495    5784 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592502    5784 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592506    5784 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592512    5784 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592516    5784 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592520    5784 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592524    5784 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592530    5784 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592533    5784 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592536    5784 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592539    5784 flags.go:64] FLAG: --cgroup-root=""
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592542    5784 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592545    5784 flags.go:64] FLAG: --client-ca-file=""
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592548    5784 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 11:51:33.598014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592553    5784 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592557    5784 flags.go:64] FLAG: --cluster-domain=""
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592560    5784 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592564    5784 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592567    5784 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592570    5784 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592593    5784 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592597    5784 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592601    5784 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592605    5784 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592608    5784 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592611    5784 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592614    5784 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592617    5784 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592620    5784 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592625    5784 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592628    5784 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592631    5784 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592634    5784 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592637    5784 flags.go:64] FLAG: --enable-server="true"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592640    5784 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592645    5784 flags.go:64] FLAG: --event-burst="100"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592649    5784 flags.go:64] FLAG: --event-qps="50"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592652    5784 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 11:51:33.598833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592656    5784 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592659    5784 flags.go:64] FLAG: --eviction-hard=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592664    5784 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592667    5784 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592670    5784 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592673    5784 flags.go:64] FLAG: --eviction-soft=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592676    5784 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592679    5784 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592688    5784 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592701    5784 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592704    5784 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592708    5784 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592710    5784 flags.go:64] FLAG: --feature-gates=""
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592715    5784 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592718    5784 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592721    5784 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592725    5784 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592728    5784 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592731    5784 flags.go:64] FLAG: --help="false"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592734    5784 flags.go:64] FLAG: --hostname-override="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592737    5784 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592740    5784 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592743    5784 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 11:51:33.599414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592746    5784 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592750    5784 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592753    5784 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592756    5784 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592758    5784 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592761    5784 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592764    5784 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592767    5784 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592770    5784 flags.go:64] FLAG: --kube-reserved=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592773    5784 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592776    5784 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592779    5784 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592782    5784 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592785    5784 flags.go:64] FLAG: --lock-file=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592788    5784 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592791    5784 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592794    5784 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592801    5784 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592804    5784 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592807    5784 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592810    5784 flags.go:64] FLAG: --logging-format="text"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592813    5784 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592816    5784 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592819    5784 flags.go:64] FLAG: --manifest-url=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592822    5784 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 11:51:33.599946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592827    5784 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592831    5784 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592835    5784 flags.go:64] FLAG: --max-pods="110"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592838    5784 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592841    5784 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592844    5784 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592847    5784 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592851    5784 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592854    5784 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592856    5784 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592864    5784 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592867    5784 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592871    5784 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592874    5784 flags.go:64] FLAG: --pod-cidr=""
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592876    5784 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592879    5784 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592882    5784 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592885    5784 flags.go:64] FLAG: --port="10250"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592888    5784 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592891    5784 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-011170f36cb618e40"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592894    5784 flags.go:64] FLAG: --qos-reserved=""
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592897    5784 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592900    5784 flags.go:64] FLAG: --register-node="true"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592903    5784 flags.go:64] FLAG: --register-with-taints=""
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592909    5784 flags.go:64] FLAG: --registry-burst="10"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592912    5784 flags.go:64] FLAG: --registry-qps="5"
Nov 03 11:51:33.600550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592915    5784 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592918    5784 flags.go:64] FLAG: --reserved-memory=""
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592921    5784 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592924    5784 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592928    5784 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592931    5784 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592933    5784 flags.go:64] FLAG: --runonce="false"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592936    5784 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592940    5784 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592943    5784 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592946    5784 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592949    5784 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592952    5784 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592955    5784 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592958    5784 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592960    5784 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592963    5784 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592966    5784 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592969    5784 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592972    5784 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592975    5784 flags.go:64] FLAG: --system-cgroups=""
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592977    5784 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592983    5784 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592985    5784 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592988    5784 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 11:51:33.601319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592994    5784 flags.go:64] FLAG: --tls-min-version=""
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.592997    5784 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593000    5784 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593003    5784 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593006    5784 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593010    5784 flags.go:64] FLAG: --v="2"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593014    5784 flags.go:64] FLAG: --version="false"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593018    5784 flags.go:64] FLAG: --vmodule=""
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593022    5784 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593026    5784 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593175    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593179    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593182    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593185    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593188    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593191    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593194    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593198    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593203    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593206    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593209    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.602005 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593211    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593214    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593216    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593219    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593222    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593225    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593227    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593230    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593232    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593235    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593237    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593241    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593244    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593247    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593249    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593252    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593255    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593258    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593260    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593263    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.602552 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593266    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593268    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593271    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593273    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593276    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593278    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593281    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593284    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593287    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593289    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593292    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593295    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593297    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593300    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593303    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593305    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593308    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593310    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593313    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593315    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.603102 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593317    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593320    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593323    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593326    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593329    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593332    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593335    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593337    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593340    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593342    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593345    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593348    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593350    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593353    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593356    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593358    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593361    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593365    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593368    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.603918 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593372    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593387    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593390    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593393    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593395    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593398    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593401    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593403    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593406    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593409    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593412    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593414    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593417    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593419    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593422    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593424    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593430    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593433    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593436    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.604554 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593438    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593441    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593443    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593446    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593448    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593451    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593453    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593456    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593458    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593461    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593464    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593466    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593469    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593471    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593474    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593476    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.605184 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593480    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.593506    5784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593545    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593550    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593553    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593555    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593559    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593561    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593564    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593567    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593570    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593572    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593575    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593577    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593582    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593585    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.605795 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593587    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593590    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593593    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593595    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593598    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593601    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593603    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593606    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593609    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593611    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593614    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593617    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593619    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593623    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593626    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593629    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593632    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593635    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593638    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.606392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593640    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593643    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593645    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593648    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593651    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593653    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593656    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593659    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593662    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593664    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593667    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593671    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593674    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593676    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593679    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593682    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593684    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593687    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593689    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.607103 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593692    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593694    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593697    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593699    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593702    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593704    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593707    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593709    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593712    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593715    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593717    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593720    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593723    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593726    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593728    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593731    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593734    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593736    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593738    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.607779 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593741    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593745    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593749    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593752    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593755    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593760    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593763    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593766    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593769    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593772    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593774    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593777    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593779    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593782    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593785    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593787    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593790    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593792    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593795    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593797    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.608261 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593800    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593802    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593805    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593808    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593811    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593813    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593816    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593819    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593821    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593824    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593827    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593830    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593832    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593835    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.593837    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.597501    5784 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.597517    5784 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.597534    5784 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.597539    5784 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598443    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.608810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598468    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598474    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598480    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598484    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598489    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598493    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598497    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598502    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598506    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598511    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598515    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598519    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598529    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598533    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598537    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598541    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598546    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598550    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598554    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598559    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.609281 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598563    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598567    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598571    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598577    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598590    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598595    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598599    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598603    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598607    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598612    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598616    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598621    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598626    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598631    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598638    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598643    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598649    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598654    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598659    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598664    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.609780 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598669    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598674    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598679    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598684    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598689    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598694    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598703    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598708    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598728    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598732    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598736    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598847    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598850    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598853    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598856    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598859    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598861    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598864    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598867    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598870    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.610313 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598874    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598878    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598881    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598884    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598887    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598890    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598892    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598895    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598898    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598900    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598903    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598906    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598908    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598911    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598914    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598917    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598919    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598922    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598924    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.610810 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598927    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598929    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598932    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598935    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598938    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598941    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598943    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598946    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598948    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598951    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598953    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598956    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598959    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598964    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598968    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598971    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598975    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598978    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598980    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.611273 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598983    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598987    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598989    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598992    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598994    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598997    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.598999    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.599036    5784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599182    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599186    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599189    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599192    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599195    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599198    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599201    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599204    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.611763 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599207    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599210    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599212    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599215    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599217    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599220    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599222    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599225    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599228    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599230    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599233    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599236    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599238    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599241    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599244    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599246    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599249    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599252    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599254    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599257    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.612149 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599259    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599262    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599265    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599267    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599270    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599272    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599274    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599277    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599279    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599282    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599285    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599287    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599290    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599293    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599295    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599298    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599301    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599303    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599306    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.612942 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599308    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599311    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599313    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599316    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599318    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599321    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599323    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599326    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599328    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599331    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599334    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599336    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599339    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599341    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599344    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599346    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599349    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599351    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599353    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.613586 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599356    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599359    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599361    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599364    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599368    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599372    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599397    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599401    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599404    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599408    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599411    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599413    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599416    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599419    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599422    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599424    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599427    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599430    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599432    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.614087 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599435    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599438    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599440    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599443    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599445    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599448    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599451    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599453    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599456    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599458    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599462    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599466    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599469    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599471    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599474    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599476    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599479    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599481    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599484    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599487    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.614555 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599489    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.599514    5784 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599548    5784 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599554    5784 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599558    5784 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599561    5784 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599563    5784 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599566    5784 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599569    5784 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599572    5784 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599574    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599576    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599579    5784 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599582    5784 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599584    5784 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599587    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 11:51:33.615029 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599589    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599592    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599594    5784 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599597    5784 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599599    5784 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599602    5784 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599604    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599608    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599610    5784 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599612    5784 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599616    5784 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599620    5784 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599623    5784 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599626    5784 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599629    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599632    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599634    5784 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599637    5784 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599639    5784 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599642    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 11:51:33.615649 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599645    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599647    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599650    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599652    5784 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599655    5784 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599657    5784 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599660    5784 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599662    5784 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599665    5784 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599667    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599669    5784 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599672    5784 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599674    5784 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599677    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599680    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599682    5784 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599685    5784 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599687    5784 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599690    5784 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599693    5784 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 11:51:33.616372 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599695    5784 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599697    5784 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599700    5784 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599702    5784 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599705    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599708    5784 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599710    5784 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599712    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599716    5784 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599718    5784 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599721    5784 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599723    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599726    5784 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599729    5784 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599731    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599734    5784 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599737    5784 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599739    5784 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599742    5784 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 11:51:33.616939 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599744    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599746    5784 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599749    5784 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599754    5784 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599757    5784 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599759    5784 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599762    5784 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599765    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599767    5784 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599770    5784 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599772    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599775    5784 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599778    5784 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599780    5784 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599783    5784 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599786    5784 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599788    5784 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599791    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599793    5784 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 11:51:33.617451 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599796    5784 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599798    5784 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599801    5784 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599804    5784 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599806    5784 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599809    5784 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599812    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599814    5784 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599817    5784 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599819    5784 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599822    5784 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599824    5784 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599827    5784 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:51:33.599832    5784 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.599892    5784 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.600886    5784 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.600934    5784 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.601110    5784 server.go:1014] "Starting client certificate rotation"
Nov 03 11:51:33.617920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.601246    5784 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.601289    5784 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 11:45:19 +0000 UTC" deadline="2028-08-10 14:49:43.407649314 +0000 UTC"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.601325    5784 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15506h58m9.80632653s"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.601579    5784 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.602654    5784 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.604223    5784 log.go:25] "Validated CRI v1 runtime API"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.606628    5784 log.go:25] "Validated CRI v1 image API"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.607527    5784 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 11:51:33.618352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.609809    5784 fs.go:135] Filesystem UUIDs: map[665cefae-866c-4ad0-abbb-d0d7bf324e05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7de24a3a-beb9-4f3b-976b-7ee4807d3db1:/dev/nvme0n1p4]
Nov 03 11:51:33.618613 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.609828    5784 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b65be0eac35cb561a5f27f92c5b29669af9de0808824a20c48e6d52252a0623/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b65be0eac35cb561a5f27f92c5b29669af9de0808824a20c48e6d52252a0623/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27aec2e03b371cfc540ac49b3e2eb09aa4dacb1b279086a360ed217f3321116d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27aec2e03b371cfc540ac49b3e2eb09aa4dacb1b279086a360ed217f3321116d/userdata/shm major:0 minor:457 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54738eeb368b15c3704971bd789516e986643a08424acc8a7232b5a7fd511c71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54738eeb368b15c3704971bd789516e986643a08424acc8a7232b5a7fd511c71/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a/userdata/shm major:0 minor:376 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc001d20f08a1eb68068443f7cd82fddea5f8af1143fc5f8d4da4841a28f4847/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc001d20f08a1eb68068443f7cd82fddea5f8af1143fc5f8d4da4841a28f4847/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00433242-908e-469a-8d10-39330e427e5a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/00433242-908e-469a-8d10-39330e427e5a/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a0c3e91-a0ba-4045-a210-240fbc41be3c/volumes/kubernetes.io~projected/kube-api-access-trtdh:{mountpoint:/var/lib/kubelet/pods/0a0c3e91-a0ba-4045-a210-240fbc41be3c/volumes/kubernetes.io~projected/kube-api-access-trtdh major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~projected/kube-api-access-z2srs:{mountpoint:/var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~projected/kube-api-access-z2srs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~secret/metrics-certs major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/alertmanager/9 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/config-reloader/6 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~empty-dir/config-out major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/kube-api-access-4p2lk:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/kube-api-access-4p2lk major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/tls-assets major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/config-volume major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/web-config major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~projected/kube-api-access-zg8zd:{mountpoint:/var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~projected/kube-api-access-zg8zd major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~projected/kube-api-access-tddxh:{mountpoint:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~projected/kube-api-access-tddxh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~projected/kube-api-access-w7qmr:{mountpoint:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~projected/kube-api-access-w7qmr major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ece716d-9af9-434f-85bf-9e6e6fb988f3/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3ece716d-9af9-434f-85bf-9e6e6fb988f3/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4616fa40-654c-46fd-9f47-aeada7820c79/volumes/kubernetes.io~projected/kube-api-access-cl2j8:{mountpoint:/var/lib/kubelet/pods/4616fa40-654c-46fd-9f47-aeada7820c79/volumes/kubernetes.io~projected/kube-api-access-cl2j8 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47b52073-13fa-4418-b8fd-f9866952b917/volumes/kubernetes.io~projected/kube-api-access-jpztv:{mountpoint:/var/lib/kubelet/pods/47b52073-13fa-4418-b8fd-f9866952b917/volumes/kubernetes.io~projected/kube-api-access-jpztv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~projected/kube-api-access-clzds:{mountpoint:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~projected/kube-api-access-clzds major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~projected/kube-api-access-j2cvj:{mountpoint:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~projected/kube-api-access-j2cvj major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~projected/kube-api-access-kbtzw:{mountpoint:/var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~projected/kube-api-access-kbtzw major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~projected/kube-api-access-lrq9z:{mountpoint:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~projected/kube-api-access-lrq9z major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~projected/kube-api-access-49wkq:{mountpoint:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~projected/kube-api-access-49wkq major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~projected/kube-api-access-5zbmq:{mountpoint:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~projected/kube-api-access-5zbmq major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~projected/kube-api-access-fh8wz:{mountpoint:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~projected/kube-api-access-fh8wz major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~projected/kube-api-access-rpbnk:{mountpoint:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~projected/kube-api-access-rpbnk major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~projected/kube-api-access-jjdv9:{mountpoint:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~projected/kube-api-access-jjdv9 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0e95d56-77bd-4a2d-b839-326a87300313/volumes/kubernetes.io~projected/kube-api-access-chqhm:{mountpoint:/var/lib/kubelet/pods/c0e95d56-77bd-4a2d-b839-326a87300313/volumes/kubernetes.io~projected/kube-api-access-chqhm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dab969bc-1102-4323-816b-eb8944cc4d82/volumes/kubernetes.io~projected/kube-api-access-fvxr5:{mountpoint:/var/lib/kubelet/pods/dab969bc-1102-4323-816b-eb8944cc4d82/volumes/kubernetes.io~projected/kube-api-access-fvxr5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~projected/kube-api-access-7h4ks:{mountpoint:/var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~projected/kube-api-access-7h4ks major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~projected/kube-api-access-vljvp:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~projected/kube-api-access-vljvp major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd4c7517-246c-40d4-84fc-bcdebf2728bb/volumes/kubernetes.io~projected/kube-api-access-djdqp:{mountpoint:/var/lib/kubelet/pods/fd4c7517-246c-40d4-84fc-bcdebf2728bb/volumes/kubernetes.io~projected/kube-api-access-djdqp major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2/volumes/kubernetes.io~projected/kube-api-access-tcb75:{mountpoint:/var/lib/kubelet/pods/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2/volumes/kubernetes.io~projected/kube-api-access-tcb75 major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c9d4ed39ea99ee23ba7deba0f0fbdac817439d0929c464878a8292b77dcc1349/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a1e79728668f4872d079e057fa874fbcdcda6cfcdced4cdbc57a460dcf24ecb1/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6e0032d0c64a34daf81c4b29dbc02c045f32f7478dbb9ec5cc3235f411a8c433/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/668d0c548c11b43b25c2b51f156f1c319b87e3b146bbac4b9129ee9a4bfcdc13/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/7a34757bc452242272096f85e54acb4dd8d7bb6cdff6e68e3394aacc0d9e2f6f/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f2a413ec75d30f587a33b65f375280022d19938c800108222acd6f23970a6466/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-125:{mountpoint:/var/lib/containers/storage/overlay/492e32b67457e5fa345cbc918edbadd18d5457c57db509fd7ed02865fb8590d5/merged major:0 minor:125 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/0f81561212c4ba8e877f20aaf31b95c5704dabc3210ec26eddbf2e9a3eb26e54/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/16330e06a686ac96f5ae4196e1c17ef1dbe5c602b54123bb3f3cd35309a03e56/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2332a59c0b8c421f99d1d437bbaeadfc68f88aefe76b70c72bca61d16b4162e7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/aa913a7572315541d793326d0e3a1591d75be0adeed7fb1e05d9e1457335e1c8/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/43b409ea016586534e60b709f9f42e267396f07e39afcaf0fe3e72952f16e9f0/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/cbb24f491f8e8fc083cf2b409df6c3fab9c15e0af24997928a62d5d52f2e451f/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/78cefcdf0e8852853ad9d25da97d6160eae351e5530b1802ddb9153e879ead4c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/af5980259f28f8996f07cb30315a81a0948e9635b7c3585b32144580b228cec9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4604a464878e544478e7d78be3311113f88596cfe5147a50f5b44e15004ad8ff/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/735ae25ab0fd902c00d6364a959f17f1d4238390acd2f05a79b47eb6597656fa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/8f417a08251c2a4be8999730d7214ed8335c1897347026f085c34200538ab8e5/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/370cb43a0642a2700f63bac7966cbeefd24a4a05fda3bf243d8fd2bf82f8d3e7/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e5acca49272fdb699996a627e42617519c2e0a401eb0b917063251d7c79ef9f0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d84582418395aacf2c8542add53d95724cb1de8d549b81418ea3cc568395fc74/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/bb6df15d5bab131f815c3289cd12cd2505f689b1dd1ea87910adb3790c162210/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6af3276c64a3e92370a6f846833d72e93c828e5186e39f01841aaef1fe63fd12/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/fd44467bdc29d5be49d061d21aa231d87a9a924697d0e423075a7744b4f4cfbb/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/d59dfe9b150a4a1a9936a8f3e3d1eb161506fdea6328033a438eeeeb500112b1/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/b2c1d7dc31a877db109665d9f741d9e3782c4d641a5b3b6e94c4560b57d2e0d7/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/970de1a9fffebd3e2c799aaaea24e298ae50810ac15d7d77246a01d976e2c681/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/75f68e46b2cb455cf4ff89af81a07a146c9d7e3d03e557110e2440b1b055e2ec/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/9fa84767b6f053bdf892d6d4e9f7f3599940c68ea42a728143d56512c9f3fcf2/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/0138c16d19b17a76cbb9f603a79fc790be7b933832899fb11ba3ff5d0cd190c0/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/180174a3295612ae17c3f60311d4ab838c4184ac8b24391519f0adbdb7d41063/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/8edcef39b1b111e35387aa903139a47e432e9f5fe5ae458f55de3a040706988b/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/2f5206acce9742ab40ff8ffe48fcba374d8b43bf451ad1826a60ca5439ffad5b/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d4bfde32435c81540edc9f0e468691128570dcb088f34a177f22817110659e7b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/21544402dcfedc80122840a38cc2e8d9a585351e6be494b71d50193494906d5f/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7ef1ac6dd74fda323a9cd1b6a0b26b5af6228cdc5af994c6f6907c736e9b9415/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/38778cffeda28316125aadf4a48ba979d58ce0d7cc5bf16ed7fa62b20c248a84/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/a4cf7a2f074f11fe3f3710332adc8f7fca9ebd2689609742476d4016ea8ffaad/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/08bfaa082f9194df00045f71216d84ca829495b4d96af1606d6be514abde20ea/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/4fbeea6043ab26791800cfbf582d476044b649a0d0f78e0139e1c6c953b483c2/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/871d52e90cf07f4c6fba4983be4e62fb65e4179b5b7a8cd58988fbbe9182e2f2/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/d07874aa194e34f1b94beaafbdbe0c8be861828bf690ff8715056e374a19f685/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/202916a9c28e11ec120eeec06152ec6d2dbed485156e7043b647e9b7a143fc2a/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/4a231b277236a54d6c3013ef05f4f3d5e4ae8b516cc72115521b91f86615117d/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/9a8bbc87f69f35f13e6bfe3b8320198cbfb367346c0c88a5e4c7fcecc1d14a1c/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/a30f3e9348022dcb2436e6d0727d4b55971afc94c9a1e13d59e27b22b0cbee29/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/5b717f1dacf4aeffa3eea42ba2a2029e55aca9519f8b4aa4c0b4101d541b4dbf/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/bbb607c38a07d0a1202d32f96125ea4aeeea18e45cc4757d6864ed7b1e54ceb7/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/5242050fc8ce93bf9a832e5137aea1def704c8f5e938c93a0a7d47f9ae3470d9/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/d58a73b53701983292fe5fa5c31fd79accea1cb502fafc99757f4ea020c0f988/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/a89d629c2ac6f811a07339e63e8b64d95eec7147e5c6816fcc1973ab9d7a2776/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/5adc7e70dc644575173d8915f5d465dc9a3d1c3b4edcb211dc0af5059fb83ef9/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/284f1a509027652367514cc9b03d610187554a6089674c0a305908bcd38aeebf/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/e77ef7c92c4ece5f8be840f1c72ae8a3fd4d713de7b110e7ab0dccb779df64fd/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/7b0bb1fa5427b7159722b00835a8fa292f64af160ebf5f69cbaa3d8efbcd401d/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/38c4a3ccec043d4a022afcf0e41b4ea23a96b666ef844220cd1ba83f7161e24f/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/a9d90d22f940a1dd1696a129afb02ab6017375f20b1b979f326c58aab3f97976/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/57f0df76d86a4837961c5179e7a6188a2eb2b892f5aef902b0fe10b03d9e48b0/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/7469efca039a9c99146571524b51cbd56b8e447b93a31aee1ef3325972eaae92/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/94299cf3f69d7cea2d8e17cf939b224ad4ff6b9a5aa8b0d65eebd75325870813/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/7ee13db5ec27e19b3e83d4ed717246a93b3d04b850b844255231e3b451342f09/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/2bba7027d465776fc3607749316410cc34ef6e9f0c78700db04234c356816e30/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/25cac8ff08491d49781ed79e6068d6f2fb25a3a74192924af93260b1e3b9bb43/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/e09c2141017910372c4726da887161044661f35c651722913eba507110faf551/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/3e7633b8d266d20ff3212f0616a5dbf450107e24f4f9b9673aa7824f46b6f23b/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/e47f59e284b1ff7314f1e13b7997d306557e0941291e30e9c8e19cf55301b701/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-467:{mountpoint:/var/lib/containers/storage/overlay/1883a861cbe0193ae96ebaf192c37dfdad9d8c7751b5205a6b81991470158bbd/merged major:0 minor:467 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/0845db8548d60246ff3dfcbe0d7794ca83a7013bb5597ca1956e63d1acf35308/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/dcd9f8215fe32087a4097f5cd7d242e7067ccffc837f7ec724d5ac29b068526d/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/a6c46c9e482aa5c4ae0a6618b29c8884ff3c3ec4299a7e2a9ead7e3196905cce/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/48a211a652691a6386a5f6afcb5d9610a5ef0d323ff955a1e46e8907c486d292/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/df767bf81406a5399813b90169921dbd35a8369efae7c446a205a7d2a23b9d71/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-494:{mountpoint:/var/lib/containers/storage/overlay/dc975d4ec899752f1fdaf531199d8e40272b983bd31ef1a04d336df6a0f1f740/merged major:0 minor:494 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/2047e7be976f57d8210084eaa7be5afefce48192c3a40c13943196fffa56ffc1/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8605dc7dc00770ba21b0c21dcf67c8768383cc136234a31e56ab1f49789e2b9c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/95a8e52a3bdb3e56463ee66e22bb370c829d226845fcbbbb6835f71c13f6bccf/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/550034268ed9512ce385e625dad450a79a05b9bcd9af80f60e44e1afc78d7243/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/01a6686e29699af31b30b76ffb18130110bbf5a5c81f1093853372cbb698856e/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/e17d535b7a5aecf5997516f5e35d99161e690554b4fd90bd51bbd319bcda0412/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a52497a7ae00d31f5e604d799ce2685f79421841c97277b4dd22eb8309dd3ab4/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/61f5de087a0b21abb67c3a7c7f4325ade9820b249d72e8d5644e9cdd0fd00901/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d387ca84d70d2a3aa642df4a4ce550da67c2c43fddc7a2fdba92268579034de2/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/717c7fd216bebba512ef832814e01afe10fbfb88287e080eee9d92e925dbe378/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 11:51:33.619402 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.618923    5784 manager.go:217] Machine: {Timestamp:2026-11-03 11:51:33.618458836 +0000 UTC m=+0.060661132 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100406 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cadd1676146352c4357b8ec6cca42 SystemUUID:ec2cadd1-6761-4635-2c43-57b8ec6cca42 BootID:cf702fac-b92b-4c0e-8cd3-6242471a95f6 Filesystems:[{Device:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~projected/kube-api-access-49wkq DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~projected/kube-api-access-zg8zd DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~projected/kube-api-access-4p2lk DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75a6f8716bff05690482ca15a2230595f28c45522496b57de80b20032037510a/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db803886e2654f2e41cc6e77b5cd3c56686f0dfdb7564154d1039f11969f5c05/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19a645e2a7a730d5560d2e7b640d552270ec812385310ef89106ac9ee0b68b64/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:409 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:298 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/deff5aafef7eb98f8b10f1dd83f7a2922a5619fc20eb986e48064635ce31b7d1/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:337 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8528159239d3b0ac5fa1a84d93fcab14abfb06de3a16c1650b9c7c6d144dbd72/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:410 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:243 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27aec2e03b371cfc540ac49b3e2eb09aa4dacb1b279086a360ed217f3321116d/userdata/shm DeviceMajor:0 DeviceMinor:457 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dab969bc-1102-4323-816b-eb8944cc4d82/volumes/kubernetes.io~projected/kube-api-access-fvxr5 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54738eeb368b15c3704971bd789516e986643a08424acc8a7232b5a7fd511c71/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~projected/kube-api-access-5zbmq DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:340 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~projected/kube-api-access-j2cvj DeviceMajor:0 DeviceMinor:411 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2/volumes/kubernetes.io~projected/kube-api-access-tcb75 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~projected/kube-api-access-kbtzw DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa70a27e1663012f93520b1f2627c30016f80d7fc555937994c5aaa15b35655a/userdata/shm DeviceMajor:0 DeviceMinor:376 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/171dc6f1251e019ba9abd125cbb348d3221fdfd7df3bfe7f2262d21811b4ea48/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0a0c3e91-a0ba-4045-a210-240fbc41be3c/volumes/kubernetes.io~projected/kube-api-access-trtdh DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~projected/kube-api-access-rpbnk DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c439b40-37df-4d6b-82ce-004d3275ad8e/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-125 DeviceMajor:0 DeviceMinor:125 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0e95d56-77bd-4a2d-b839-326a87300313/volumes/kubernetes.io~projected/kube-api-access-chqhm DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~projected/kube-api-access-vljvp DeviceMajor:0 DeviceMinor:379 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa6cd7d1e64cd6ed602be21c4458da12582dd76e1e9d8cd9375c298657c17a8b/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~projected/kube-api-access-w7qmr DeviceMajor:0 DeviceMinor:447 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f396435d39226243f2dadf89cead242c8e30d326bad8d774562de6b4c3949759/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ba0a7a61973141657d519093ef4138c6826d52b1344daaadb66ddd5cb0684e9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~projected/kube-api-access-jjdv9 DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:372 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~projected/kube-api-access-clzds DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b7205a02276ae0f826981405e6fad91c31b2500d652116e385c6582ca3fcf92/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9ab03b1374770e6e41660905c1682d3dad2c9c89ae72745e1055d49549309cc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/200522d1-e408-41f4-be88-15c3600c7225/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1d64116a2490dbfd8925b5d5e1dc1605e205d1fa929cb068125cfbe00a9e4cc8/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71023ac9a6e4c21f801cde43e95b2d9e0479e0cdf324a5f6196005e41ed7bfea/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd4c7517-246c-40d4-84fc-bcdebf2728bb/volumes/kubernetes.io~projected/kube-api-access-djdqp DeviceMajor:0 DeviceMinor:328 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a36293e5-2420-4631-ad8b-5747f97896f1/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:299 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/333cafe9cd75d15e2671bc0a85ac423341e531f6121da92db3515733c55dbe61/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-494 DeviceMajor:0 DeviceMinor:494 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~projected/kube-api-access-7h4ks DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~projected/kube-api-access-lrq9z DeviceMajor:0 DeviceMinor:296 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:370 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/49f19698-00a5-4e9e-a7c3-3fd1a8742421/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:405 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/488fc93c-6c34-49ae-83b9-9e6f930ad2b7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5025802c-cc81-4a98-8248-8878e9f36e10/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f14e9e71-6399-459d-9ade-004b2390424a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:371 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35f05625ca1986332102ba12652d00e3a562b5f6c02079431cf68ec9004038f5/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/47b52073-13fa-4418-b8fd-f9866952b917/volumes/kubernetes.io~projected/kube-api-access-jpztv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4729103a9ca08ad8ada9356aa4759eae1a021b758b181c8dd71bce950a96a42/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e91a382-0182-42f7-aa04-29c13b922a49/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:282 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b65be0eac35cb561a5f27f92c5b29669af9de0808824a20c48e6d52252a0623/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:335 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5605fa-4135-4de2-a8ab-1d984d5c51be/volumes/kubernetes.io~projected/kube-api-access-z2srs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:244 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/4616fa40-654c-46fd-9f47-aeada7820c79/volumes/kubernetes.io~projected/kube-api-access-cl2j8 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc001d20f08a1eb68068443f7cd82fddea5f8af1143fc5f8d4da4841a28f4847/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:336 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/00433242-908e-469a-8d10-39330e427e5a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ece716d-9af9-434f-85bf-9e6e6fb988f3/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:327 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:445 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-467 DeviceMajor:0 DeviceMinor:467 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volumes/kubernetes.io~projected/kube-api-access-tddxh DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ee3aadf1-19c3-41bc-b242-6230cfcac9c3/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7aaede73-1b46-4d67-a336-377d44b8c286/volumes/kubernetes.io~projected/kube-api-access-fh8wz DeviceMajor:0 DeviceMinor:245 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0b65be0eac35cb5 MacAddress:32:10:37:5d:1a:42 Speed:10000 Mtu:8901} {Name:171dc6f1251e019 MacAddress:ba:05:db:75:84:4b Speed:10000 Mtu:8901} {Name:19a645e2a7a730d MacAddress:22:af:6e:af:29:74 Speed:10000 Mtu:8901} {Name:27aec2e03b371cf MacAddress:62:1d:bc:f9:d2:c2 Speed:10000 Mtu:8901} {Name:333cafe9cd75d15 MacAddress:a6:bc:c6:a1:fb:4a Speed:10000 Mtu:8901} {Name:54738eeb368b15c MacAddress:d2:ff:d0:c2:67:f6 Speed:10000 Mtu:8901} {Name:666137b783eeee3 MacAddress:de:b0:c0:4e:86:fa Speed:10000 Mtu:8901} {Name:75a6f8716bff056 MacAddress:6a:74:31:de:09:17 Speed:10000 Mtu:8901} {Name:96c8c862425e781 MacAddress:ae:ae:8c:ed:ce:17 Speed:10000 Mtu:8901} {Name:9b7205a02276ae0 MacAddress:d2:78:9c:50:09:3c Speed:10000 Mtu:8901} {Name:aa70a27e1663012 MacAddress:2e:95:b0:1a:09:1a Speed:10000 Mtu:8901} {Name:b56a7c86bc6594d MacAddress:e6:1e:58:07:e1:06 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:dc:b6:f2:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:fa:15:90:88:2e:6d Speed:0 Mtu:8901} {Name:cc001d20f08a1eb MacAddress:da:04:b7:bd:c1:e2 Speed:10000 Mtu:8901} {Name:db803886e2654f2 MacAddress:fe:b5:ee:fc:81:a1 Speed:10000 Mtu:8901} {Name:deff5aafef7eb98 MacAddress:9e:02:2d:2d:9a:79 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:dc:b6:f2:dd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c2:db:e8:8a:90:81 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:77:e7:f1:73:7f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 11:51:33.619402 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619403    5784 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 11:51:33.619650 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619451    5784 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 11:51:33.619759 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619729    5784 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 11:51:33.619888 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619761    5784 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-140.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 11:51:33.619934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619890    5784 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 11:51:33.619934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619899    5784 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 11:51:33.619934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619909    5784 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:51:33.619934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.619922    5784 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 11:51:33.620053 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620019    5784 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 11:51:33.620179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620168    5784 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 11:51:33.620297 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620286    5784 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 11:51:33.620345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620334    5784 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 11:51:33.620390 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620354    5784 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 11:51:33.620390 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620366    5784 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 11:51:33.620466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620397    5784 kubelet.go:405] "Adding apiserver pod source"
Nov 03 11:51:33.620466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.620407    5784 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 11:51:33.621103 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.621088    5784 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:51:33.621103 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.621100    5784 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:51:33.621604 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.621586    5784 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 11:51:33.624187 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624172    5784 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 11:51:33.624187 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624186    5784 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 11:51:33.624299 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624280    5784 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 11:51:33.624514 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624502    5784 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 11:51:33.624549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624532    5784 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 11:51:33.624632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624623    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624637    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624644    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624650    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624656    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624662    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624667    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 11:51:33.624671 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624673    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624681    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624687    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624703    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624712    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624741    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 11:51:33.624902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.624747    5784 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 11:51:33.628115 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.628103    5784 server.go:1263] "Started kubelet"
Nov 03 11:51:33.628210 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.628165    5784 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 11:51:33.628670 ip-10-0-139-140 systemd[1]: Started Kubernetes Kubelet.
Nov 03 11:51:33.629455 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.628730    5784 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 11:51:33.629455 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.629285    5784 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 11:51:33.629760 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.629666    5784 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 11:51:33.630730 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.630715    5784 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.635298    5784 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.635461    5784 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.635475    5784 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.635530    5784 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.635711    5784 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636105    5784 factory.go:55] Registering systemd factory
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636123    5784 factory.go:223] Registration of the systemd container factory successfully
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636407    5784 factory.go:153] Registering CRI-O factory
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636422    5784 factory.go:223] Registration of the crio container factory successfully
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636466    5784 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636480    5784 factory.go:103] Registering Raw factory
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636492    5784 manager.go:1199] Started watching for new ooms in manager
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.636747    5784 manager.go:319] Starting recovery of all containers
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639419    5784 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639481    5784 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 11:45:21 +0000 UTC" deadline="2028-05-31 05:46:28.574686689 +0000 UTC"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639500    5784 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13793h54m54.935189683s"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639630    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639664    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639678    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639692    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3aadf1-19c3-41bc-b242-6230cfcac9c3" volumeName="kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639707    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00433242-908e-469a-8d10-39330e427e5a" volumeName="kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639721    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639734    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639746    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72" volumeName="kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639759    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e91a382-0182-42f7-aa04-29c13b922a49" volumeName="kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639768    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639775    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaede73-1b46-4d67-a336-377d44b8c286" volumeName="kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639785    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639793    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639806    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639823    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488fc93c-6c34-49ae-83b9-9e6f930ad2b7" volumeName="kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639837    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639850    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639863    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639876    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639889    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639902    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ece716d-9af9-434f-85bf-9e6e6fb988f3" volumeName="kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639916    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639928    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaede73-1b46-4d67-a336-377d44b8c286" volumeName="kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639943    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639956    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd84fe1e-ce28-4634-b84c-fa50c5dc2be2" volumeName="kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639969    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd84fe1e-ce28-4634-b84c-fa50c5dc2be2" volumeName="kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639982    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00433242-908e-469a-8d10-39330e427e5a" volumeName="kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.639995    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0c3e91-a0ba-4045-a210-240fbc41be3c" volumeName="kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640014    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200522d1-e408-41f4-be88-15c3600c7225" volumeName="kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640045    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e91a382-0182-42f7-aa04-29c13b922a49" volumeName="kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640073    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e91a382-0182-42f7-aa04-29c13b922a49" volumeName="kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640086    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640099    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640114    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640128    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640141    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488fc93c-6c34-49ae-83b9-9e6f930ad2b7" volumeName="kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640152    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5025802c-cc81-4a98-8248-8878e9f36e10" volumeName="kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640159    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640167    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaede73-1b46-4d67-a336-377d44b8c286" volumeName="kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640174    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640181    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640189    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a0c3e91-a0ba-4045-a210-240fbc41be3c" volumeName="kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640196    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640204    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640212    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47b52073-13fa-4418-b8fd-f9866952b917" volumeName="kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640220    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="488fc93c-6c34-49ae-83b9-9e6f930ad2b7" volumeName="kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640227    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e91a382-0182-42f7-aa04-29c13b922a49" volumeName="kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640240    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7aaede73-1b46-4d67-a336-377d44b8c286" volumeName="kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640249    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640256    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5605fa-4135-4de2-a8ab-1d984d5c51be" volumeName="kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640264    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640272    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47b52073-13fa-4418-b8fd-f9866952b917" volumeName="kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640279    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640290    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab969bc-1102-4323-816b-eb8944cc4d82" volumeName="kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640300    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640312    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640335    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640343    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e95d56-77bd-4a2d-b839-326a87300313" volumeName="kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640350    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab969bc-1102-4323-816b-eb8944cc4d82" volumeName="kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640358    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dab969bc-1102-4323-816b-eb8944cc4d82" volumeName="kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640366    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3aadf1-19c3-41bc-b242-6230cfcac9c3" volumeName="kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640388    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640396    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640404    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640411    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3aadf1-19c3-41bc-b242-6230cfcac9c3" volumeName="kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640419    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640426    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd4c7517-246c-40d4-84fc-bcdebf2728bb" volumeName="kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640433    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640440    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640448    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640455    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4616fa40-654c-46fd-9f47-aeada7820c79" volumeName="kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640474    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5025802c-cc81-4a98-8248-8878e9f36e10" volumeName="kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640481    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640488    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640495    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640515    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640523    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640532    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72" volumeName="kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640540    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640550    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c439b40-37df-4d6b-82ce-004d3275ad8e" volumeName="kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640562    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b543b127-aa97-4f98-863e-ca16072e4031" volumeName="kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640570    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640577    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640585    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200522d1-e408-41f4-be88-15c3600c7225" volumeName="kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640592    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200522d1-e408-41f4-be88-15c3600c7225" volumeName="kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640599    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640607    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640614    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640621    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640629    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72" volumeName="kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640639    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4616fa40-654c-46fd-9f47-aeada7820c79" volumeName="kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640650    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640657    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee3aadf1-19c3-41bc-b242-6230cfcac9c3" volumeName="kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640665    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640672    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14e9e71-6399-459d-9ade-004b2390424a" volumeName="kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640680    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5605fa-4135-4de2-a8ab-1d984d5c51be" volumeName="kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640687    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="200522d1-e408-41f4-be88-15c3600c7225" volumeName="kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640704    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640716    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640727    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47b52073-13fa-4418-b8fd-f9866952b917" volumeName="kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640735    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640764    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" volumeName="kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640773    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640782    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c41b69d-ee39-45cb-8a06-66b5029f2952" volumeName="kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640790    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72" volumeName="kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640797    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72" volumeName="kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640805    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4550a1-3f78-4685-aa82-2546c2aa7b41" volumeName="kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640813    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47b52073-13fa-4418-b8fd-f9866952b917" volumeName="kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640820    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49f19698-00a5-4e9e-a7c3-3fd1a8742421" volumeName="kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640827    5784 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36293e5-2420-4631-ad8b-5747f97896f1" volumeName="kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile" seLinuxMountContext=""
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640835    5784 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 11:51:33.641045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.640839    5784 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 11:51:33.672147 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.672108    5784 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 11:51:33.673661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.673626    5784 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 11:51:33.673661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.673652    5784 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 11:51:33.673864 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.673677    5784 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 11:51:33.673864 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.673720    5784 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 11:51:33.686746 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.686714    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2" exitCode=0
Nov 03 11:51:33.703232 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.703199    5784 generic.go:356] "Generic (PLEG): container finished" podID="a36293e5-2420-4631-ad8b-5747f97896f1" containerID="1103a83aae7ff99b4eeabf4cc2f3906ed3655e7ce9f8813baf93cb85ca4fb837" exitCode=0
Nov 03 11:51:33.713660 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713621    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="6ffd97eeda26ae86779a494b029d8751b9961e01634ffd71f124c52e11f120c1" exitCode=0
Nov 03 11:51:33.713660 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713654    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d26218639fcb239885b8bc00f6a579b6f935d908a50282482e43d780a4743fd7" exitCode=0
Nov 03 11:51:33.713660 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713666    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="65a962bbfcfca4e91e0bc7571ca6f0c06d97e11fec39f365df57f4a2fb2466cc" exitCode=0
Nov 03 11:51:33.713886 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713676    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="29c906b063361d5719b3505fc4548ce796e1bda0de85345256cb633a36dc8ad3" exitCode=0
Nov 03 11:51:33.713886 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713686    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d90518b10c6adbbe7c3c8f497542122a546f0617a9e9fb08dea0857b494fe7cb" exitCode=0
Nov 03 11:51:33.713886 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.713699    5784 generic.go:356] "Generic (PLEG): container finished" podID="47b52073-13fa-4418-b8fd-f9866952b917" containerID="d73e6682745ed6f0797339fba701ddfd784ced035a7056674a0880ad3824682d" exitCode=0
Nov 03 11:51:33.715546 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.715513    5784 generic.go:356] "Generic (PLEG): container finished" podID="71fada2c7443a601ca789370ef919cac" containerID="2c3e2c05753628746b8f432f622c07abb25412436c2f571fc4226c358bc7e718" exitCode=0
Nov 03 11:51:33.735841 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.735768    5784 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:51:33.770720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.770699    5784 manager.go:324] Recovery completed
Nov 03 11:51:33.775339 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.775230    5784 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 11:51:33.795740 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.795710    5784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:51:33.797563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.797542    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:51:33.797563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.797578    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:51:33.797563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.797589    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:51:33.800205 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800170    5784 cpu_manager.go:226] "Starting" policy="none"
Nov 03 11:51:33.800205 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800187    5784 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 11:51:33.800205 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800187    5784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:51:33.800433 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800209    5784 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 11:51:33.800433 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800419    5784 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800436    5784 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800460    5784 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800469    5784 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800477    5784 policy_none.go:50] "Start"
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800489    5784 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 11:51:33.800530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800504    5784 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 11:51:33.800842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800686    5784 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 11:51:33.800842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800705    5784 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 11:51:33.800842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.800713    5784 policy_none.go:44] "Start"
Nov 03 11:51:33.804719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.804703    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:51:33.804791 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.804730    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:51:33.804791 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.804740    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:51:33.805562 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.805537    5784 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 11:51:33.810717 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.810705    5784 manager.go:341] "Starting Device Plugin manager"
Nov 03 11:51:33.810797 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.810728    5784 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 11:51:33.810797 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.810737    5784 server.go:87] "Starting device plugin registration server"
Nov 03 11:51:33.810987 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.810969    5784 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 11:51:33.811093 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.810984    5784 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 11:51:33.811093 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.811049    5784 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 11:51:33.811184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.811148    5784 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 11:51:33.811231 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.811185    5784 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 11:51:33.811231 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.811191    5784 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 11:51:33.811319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.811241    5784 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T11:51:33.811174714Z","UUID":"db73439c-d834-40a0-9954-d4a6e931a913","Handler":null,"Name":"","Endpoint":""}
Nov 03 11:51:33.813126 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.813094    5784 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 11:51:33.813231 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:33.813138    5784 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-140.us-west-2.compute.internal\" not found"
Nov 03 11:51:33.814282 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.814262    5784 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 11:51:33.814397 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.814292    5784 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 11:51:33.816497 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.816473    5784 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:51:33.816585 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.816474    5784 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:51:33.816661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.816646    5784 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 11:51:33.863969 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.863944    5784 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:51:33.908573 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.908545    5784 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 11:51:33.911936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.911920    5784 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 11:51:33.912956 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.912937    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 11:51:33.913029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.912966    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 11:51:33.913029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.912976    5784 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-140.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 11:51:33.913029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.913021    5784 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:33.976008 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.975963    5784 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"]
Nov 03 11:51:33.976607 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976571    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"84ff2cd276a94f8c36394a4ab321034f","Type":"ContainerStarted","Data":"afb9eda708279ee84ea94d0c08b02b8ce57ec5117c95e75caaff4cac7a35befa"}
Nov 03 11:51:33.976710 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976617    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"84ff2cd276a94f8c36394a4ab321034f","Type":"ContainerStarted","Data":"61ba2664600e0efa2fcd0367097d628db2ded1a8a6cda327fe1ca8e99c3e864a"}
Nov 03 11:51:33.976710 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976661    5784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:33.976805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976715    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerStarted","Data":"5c99352e39372fae136f24732b55bbcf2b89f2517b133597f7a80ec8a32c3a0f"}
Nov 03 11:51:33.976805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976730    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerDied","Data":"2c3e2c05753628746b8f432f622c07abb25412436c2f571fc4226c358bc7e718"}
Nov 03 11:51:33.976805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976741    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal" event={"ID":"71fada2c7443a601ca789370ef919cac","Type":"ContainerStarted","Data":"96496b7694824f03941161a07056aca31a640d017d299e44af55ed04294ef5f0"}
Nov 03 11:51:33.976940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:33.976898    5784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.013686 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.013628    5784 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.013763 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.013692    5784 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.037084 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.037059    5784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:51:34.037177 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:34.037103    5784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.039622 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.039606    5784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:51:34.039659 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:34.039644    5784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.117950 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.117919    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.118126 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.117951    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.118126 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.117974    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218609 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218574    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218609 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218605    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218623    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218672    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84ff2cd276a94f8c36394a4ab321034f-config\") pod \"kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"84ff2cd276a94f8c36394a4ab321034f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218691    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.218784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.218691    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71fada2c7443a601ca789370ef919cac-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" (UID: \"71fada2c7443a601ca789370ef919cac\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.620804 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.620776    5784 apiserver.go:52] "Watching apiserver"
Nov 03 11:51:34.747707 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.747674    5784 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.794901 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.794866    5784 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 11:51:34.797037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.796873    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal","openshift-console/console-86dfb58d96-wn4hk","openshift-ingress-canary/ingress-canary-mw5bn","openshift-monitoring/metrics-server-96685b96b-5s9hm","openshift-monitoring/node-exporter-wrh8c","kube-system/kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-m2svl","openshift-monitoring/thanos-querier-7748596f65-48dhl","openshift-multus/multus-additional-cni-plugins-7jh9h","openshift-multus/network-metrics-daemon-r84qx","kube-system/konnectivity-agent-785td","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp","openshift-image-registry/node-ca-6htsx","openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh","kube-system/global-pull-secret-syncer-8tx9f","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8","openshift-network-operator/iptables-alerter-l4fvp","openshift-console/console-bbf444864-cm8fl","openshift-multus/multus-4gqzk","openshift-ovn-kubernetes/ovnkube-node-b7h7z","openshift-console/console-5db5589b4-9v2nm","openshift-insights/insights-runtime-extractor-gk9d8","openshift-network-diagnostics/network-check-target-58msl","openshift-dns/dns-default-jhqfb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s","openshift-dns/node-resolver-dpmg7"]
Nov 03 11:51:34.803439 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.803420    5784 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 11:51:34.803549 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:34.803461    5784 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-140.us-west-2.compute.internal"
Nov 03 11:51:34.838243 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.838215    5784 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 11:51:34.846973 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.846943    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.846973 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.846950    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.846984    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.846977    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xf9nj\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.846958    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.847011    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.847034    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxdwl\""
Nov 03 11:51:34.847179 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.847083    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.847459 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.847432    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.848287 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848264    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g5g7l\""
Nov 03 11:51:34.848428 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848408    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.848504 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848454    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 11:51:34.848504 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848490    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 11:51:34.848661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848644    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jp4k\""
Nov 03 11:51:34.848832 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848811    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 11:51:34.848943 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848921    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 11:51:34.849003 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.848994    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.849172 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849154    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.849224 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849205    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 11:51:34.849477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849458    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.849538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849515    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.849605 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849596    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.849697 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849680    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 11:51:34.849794 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849778    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xf5fg\""
Nov 03 11:51:34.849932 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849913    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-65pss\""
Nov 03 11:51:34.849932 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849928    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.850056 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.849928    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vn7bf\""
Nov 03 11:51:34.850672 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.850656    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.851189 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851165    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 11:51:34.851279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851263    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 11:51:34.851370 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851355    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 11:51:34.851488 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851467    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.851488 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851484    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.851593 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851506    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-swh4m\""
Nov 03 11:51:34.851593 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851538    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.851593 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851578    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jp68j\""
Nov 03 11:51:34.851719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851650    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6gwtx\""
Nov 03 11:51:34.851719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851688    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zm8xm\""
Nov 03 11:51:34.851719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851711    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 11:51:34.851842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.851763    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 11:51:34.890229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890203    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x96q5\""
Nov 03 11:51:34.890422 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890403    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 11:51:34.890592 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890570    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 11:51:34.890699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890641    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 11:51:34.890699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890649    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 11:51:34.890699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890693    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 11:51:34.890874 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890713    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 11:51:34.890936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890913    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 11:51:34.890987 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890942    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 11:51:34.890987 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890953    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 11:51:34.890987 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890955    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9vqqu7n71jhmb\""
Nov 03 11:51:34.890987 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890943    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.890993    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-46zdg\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891007    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891093    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891120    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a2tbc0bg4loul\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891145    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-phz8z\""
Nov 03 11:51:34.891186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891174    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n7npm\""
Nov 03 11:51:34.891350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891250    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nhpqx\""
Nov 03 11:51:34.891350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891312    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jslfh\""
Nov 03 11:51:34.891350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.891339    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzgk2\""
Nov 03 11:51:34.922363 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922344    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 11:51:34.922472 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922397    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.922472 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922439    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 11:51:34.922582 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922493    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 11:51:34.922582 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922540    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.922666 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922598    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.922666 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922608    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 11:51:34.922783 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922769    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.922824 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922804    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 11:51:34.922911 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922895    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.922973 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922961    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.923023 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.922978    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 11:51:34.923172 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923157    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 11:51:34.923172 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923164    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923198    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923228    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923250    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923254    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923279    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.923331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923285    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923333    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923415    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923425    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923350    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqhm\" (UniqueName: \"kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923463    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/200522d1-e408-41f4-be88-15c3600c7225-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923585    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-tls\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.923632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923611    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923657    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923691    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923721    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923747    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923796    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923826    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923845    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923867    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.923948 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923891    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923979    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6c439b40-37df-4d6b-82ce-004d3275ad8e-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.923998    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvxr5\" (UniqueName: \"kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924020    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924047    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-tuned\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924085    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-config\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924139    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924139    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.924279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924183    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924280    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924303    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924320    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924336    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924353    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924368    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924406    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924426    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924461    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/200522d1-e408-41f4-be88-15c3600c7225-data-volume\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924500    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924567    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924582    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e91a382-0182-42f7-aa04-29c13b922a49-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924591    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-metrics-client-ca\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.924599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924603    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924630    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924613    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924686    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924719    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924736    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924755    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924771    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924796    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924814    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-multus-daemon-config\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924823    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcb75\" (UniqueName: \"kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924823    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4616fa40-654c-46fd-9f47-aeada7820c79-iptables-alerter-script\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924846    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924864    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-metrics-server-audit-profiles\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924872    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924909    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924906    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49f19698-00a5-4e9e-a7c3-3fd1a8742421-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924986    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.924995    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-metrics-tls\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925030    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925036    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-tmp-dir\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925074    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-client-certs\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925068    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925112    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925117    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925130    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925151    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925226    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925252    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-binary-copy\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925262    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:34.925272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925284    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925304    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925400    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925422    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925438    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925453    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925483    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a5605fa-4135-4de2-a8ab-1d984d5c51be-metrics-certs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925474    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925522    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925535    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925543    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925555    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-tmp\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925562    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925602    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925627    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925634    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00433242-908e-469a-8d10-39330e427e5a-konnectivity-ca\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925675    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925706    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925728    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-config-volume\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925742    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovnkube-script-lib\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925731    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925742    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925858    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925876    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925899    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925901    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925936    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925962    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8wz\" (UniqueName: \"kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.925988    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926008    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926025    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926042    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926064    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926086    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926115    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926134    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926177    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926205    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926216    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926302    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.926344 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926336    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926372    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926467    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926492    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926513    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926528    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljvp\" (UniqueName: \"kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926543    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926584    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-secret-metrics-server-tls\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926559    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzds\" (UniqueName: \"kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926613    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/49f19698-00a5-4e9e-a7c3-3fd1a8742421-audit-log\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926630    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926660    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926685    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926710    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926736    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926762    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926788    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926837    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtdh\" (UniqueName: \"kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926873    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpztv\" (UniqueName: \"kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926899    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrq9z\" (UniqueName: \"kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926925    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926952    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926966    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.926976    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927001    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tddxh\" (UniqueName: \"kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927043    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbtzw\" (UniqueName: \"kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927091    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927118    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927154    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927182    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927204    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927211    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927210    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-env-overrides\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927239    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl2j8\" (UniqueName: \"kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927269    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927271    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927315    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927341    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927396    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927427    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2srs\" (UniqueName: \"kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927443    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927452    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927479    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927579    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927583    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927637    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpbnk\" (UniqueName: \"kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927669    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.927699 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927695    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927729    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927757    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927796    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927818    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927866    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927899    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927939    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927979    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8zd\" (UniqueName: \"kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.927998    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928117    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928134    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928158    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928188    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928207    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/49f19698-00a5-4e9e-a7c3-3fd1a8742421-client-ca-bundle\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928215    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928247    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928265    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928280    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928303    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4ks\" (UniqueName: \"kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928320    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928336    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928373    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928419    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928483    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928514    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00433242-908e-469a-8d10-39330e427e5a-agent-certs\") pod \"konnectivity-agent-785td\" (UID: \"00433242-908e-469a-8d10-39330e427e5a\") " pod="kube-system/konnectivity-agent-785td"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928561    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928582    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/47b52073-13fa-4418-b8fd-f9866952b917-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928598    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928630    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928637    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928663    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928678    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928717    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-ovn-node-metrics-cert\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928771    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928790    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928817    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928873    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928897    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928926    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928946    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.928981    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929006    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-textfile\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929036    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.929092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929108    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929141    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a0c3e91-a0ba-4045-a210-240fbc41be3c-serviceca\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929165    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929209    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbmq\" (UniqueName: \"kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929253    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929273    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f14e9e71-6399-459d-9ade-004b2390424a-metrics-client-ca\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929288    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929322    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f14e9e71-6399-459d-9ade-004b2390424a-tmp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929322    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929348    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929408    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929485    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929505    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929509    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dab969bc-1102-4323-816b-eb8944cc4d82-cni-binary-copy\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929524    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cvj\" (UniqueName: \"kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929541    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929558    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929574    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929604    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929632    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929658    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929658    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929688    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929703    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929737    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929776    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929780    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-tmp-dir\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929820    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929846    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.929978    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.930034    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-grpc-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.930801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.930402    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.934445 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.934429    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7aaede73-1b46-4d67-a336-377d44b8c286-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.936297 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.936279    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 11:51:34.936485 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.936413    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 11:51:34.937638 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.937618    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.937843 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.937830    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.946017 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.945999    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.980267 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.980234    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.980478 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.980460    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqhm\" (UniqueName: \"kubernetes.io/projected/c0e95d56-77bd-4a2d-b839-326a87300313-kube-api-access-chqhm\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:34.980641 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.980627    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvxr5\" (UniqueName: \"kubernetes.io/projected/dab969bc-1102-4323-816b-eb8944cc4d82-kube-api-access-fvxr5\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:34.980846 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.980831    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcb75\" (UniqueName: \"kubernetes.io/projected/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-kube-api-access-tcb75\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:34.986275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.986255    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbtzw\" (UniqueName: \"kubernetes.io/projected/5025802c-cc81-4a98-8248-8878e9f36e10-kube-api-access-kbtzw\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:34.986759 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.986740    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpbnk\" (UniqueName: \"kubernetes.io/projected/a36293e5-2420-4631-ad8b-5747f97896f1-kube-api-access-rpbnk\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.987354 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987218    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4ks\" (UniqueName: \"kubernetes.io/projected/ee3aadf1-19c3-41bc-b242-6230cfcac9c3-kube-api-access-7h4ks\") pod \"dns-default-jhqfb\" (UID: \"ee3aadf1-19c3-41bc-b242-6230cfcac9c3\") " pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:34.987354 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987299    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbmq\" (UniqueName: \"kubernetes.io/projected/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-api-access-5zbmq\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.987354 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987299    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl2j8\" (UniqueName: \"kubernetes.io/projected/4616fa40-654c-46fd-9f47-aeada7820c79-kube-api-access-cl2j8\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:34.987569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987549    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.987628 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987579    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljvp\" (UniqueName: \"kubernetes.io/projected/f14e9e71-6399-459d-9ade-004b2390424a-kube-api-access-vljvp\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.987628 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987620    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2srs\" (UniqueName: \"kubernetes.io/projected/1a5605fa-4135-4de2-a8ab-1d984d5c51be-kube-api-access-z2srs\") pod \"network-metrics-daemon-r84qx\" (UID: \"1a5605fa-4135-4de2-a8ab-1d984d5c51be\") " pod="openshift-multus/network-metrics-daemon-r84qx"
Nov 03 11:51:34.987720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.987626    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tddxh\" (UniqueName: \"kubernetes.io/projected/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-kube-api-access-tddxh\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:34.988183 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988159    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtdh\" (UniqueName: \"kubernetes.io/projected/0a0c3e91-a0ba-4045-a210-240fbc41be3c-kube-api-access-trtdh\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:34.988414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988393    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpztv\" (UniqueName: \"kubernetes.io/projected/47b52073-13fa-4418-b8fd-f9866952b917-kube-api-access-jpztv\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:34.988499 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988417    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8wz\" (UniqueName: \"kubernetes.io/projected/7aaede73-1b46-4d67-a336-377d44b8c286-kube-api-access-fh8wz\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.988705 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988678    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.988758 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988683    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.988758 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988720    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cvj\" (UniqueName: \"kubernetes.io/projected/49f19698-00a5-4e9e-a7c3-3fd1a8742421-kube-api-access-j2cvj\") pod \"metrics-server-96685b96b-5s9hm\" (UID: \"49f19698-00a5-4e9e-a7c3-3fd1a8742421\") " pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:34.988853 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988832    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8zd\" (UniqueName: \"kubernetes.io/projected/200522d1-e408-41f4-be88-15c3600c7225-kube-api-access-zg8zd\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.988953 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.988940    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzds\" (UniqueName: \"kubernetes.io/projected/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-kube-api-access-clzds\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:34.989049 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.989034    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrq9z\" (UniqueName: \"kubernetes.io/projected/5e91a382-0182-42f7-aa04-29c13b922a49-kube-api-access-lrq9z\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.989080 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.989070    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdqp\" (UniqueName: \"kubernetes.io/projected/fd4c7517-246c-40d4-84fc-bcdebf2728bb-kube-api-access-djdqp\") pod \"network-check-target-58msl\" (UID: \"fd4c7517-246c-40d4-84fc-bcdebf2728bb\") " pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:34.990292 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990242    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 11:51:34.990292 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990257    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 11:51:34.990292 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990275    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 11:51:34.990292 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990288    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 11:51:34.990549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990275    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 11:51:34.990549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990364    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 11:51:34.990549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990516    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990544    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8zh8\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990567    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990587    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990588    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990611    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990626    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2ks4w\""
Nov 03 11:51:34.990694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990674    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 11:51:34.990891 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990714    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 11:51:34.990891 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990738    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 11:51:34.990891 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.990800    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 11:51:34.991483 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.991462    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sfhxq\""
Nov 03 11:51:34.991549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.991515    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nxjhp\""
Nov 03 11:51:34.991549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.991520    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k6jvk\""
Nov 03 11:51:34.991549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.991521    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 11:51:34.991713 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.991693    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 11:51:34.995049 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.994996    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.995476 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995447    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5025802c-cc81-4a98-8248-8878e9f36e10-cert\") pod \"ingress-canary-mw5bn\" (UID: \"5025802c-cc81-4a98-8248-8878e9f36e10\") " pod="openshift-ingress-canary/ingress-canary-mw5bn"
Nov 03 11:51:34.995476 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995456    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-tls\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.995569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995451    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.995569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995450    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/200522d1-e408-41f4-be88-15c3600c7225-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:34.995569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995472    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.995569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.995552    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"console-bbf444864-cm8fl\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") " pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:34.996171 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.996152    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"console-86dfb58d96-wn4hk\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") " pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:34.996513 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.996497    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.997603 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.997586    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7aaede73-1b46-4d67-a336-377d44b8c286-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-dm9h8\" (UID: \"7aaede73-1b46-4d67-a336-377d44b8c286\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-dm9h8"
Nov 03 11:51:34.997670 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.997654    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:34.997707 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.997657    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c439b40-37df-4d6b-82ce-004d3275ad8e-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fmr2s\" (UID: \"6c439b40-37df-4d6b-82ce-004d3275ad8e\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fmr2s"
Nov 03 11:51:34.998804 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.998788    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"console-5db5589b4-9v2nm\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") " pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:51:34.998867 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.998790    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.998901 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.998869    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.998998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.998984    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:51:34.999502 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.999487    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f14e9e71-6399-459d-9ade-004b2390424a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7748596f65-48dhl\" (UID: \"f14e9e71-6399-459d-9ade-004b2390424a\") " pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:34.999605 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.999591    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e91a382-0182-42f7-aa04-29c13b922a49-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pmqqh\" (UID: \"5e91a382-0182-42f7-aa04-29c13b922a49\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pmqqh"
Nov 03 11:51:34.999669 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:34.999634    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3ece716d-9af9-434f-85bf-9e6e6fb988f3-original-pull-secret\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:35.007485 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.007452    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:35.030971 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.030946    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.030971 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.030971    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.030988    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031003    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031041    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-k8s-cni-cncf-io\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031063    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031062    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-node-log\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031077    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.031099 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031098    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-sys\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031113    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031140    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-bin\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031142    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031161    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-host\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031148    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031188    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-node-exporter-wtmp\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031191    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-sys\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031190    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031216    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-etc-kubernetes\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031222    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031237    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031252    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031273    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031284    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-dbus\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:35.031295 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031287    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031297    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-socket-dir-parent\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031311    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-kubernetes\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031320    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031353    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-kubelet\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031369    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031394    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-hostroot\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031370    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-cnibin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031412    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031435    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-registration-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031459    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031483    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031491    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031496    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031499    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031515    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-ovn\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031519    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031536    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031544    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysconfig\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031521    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-cni-netd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031562    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031576    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-system-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031581    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031597    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031613    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031613    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-device-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031633    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a0c3e91-a0ba-4045-a210-240fbc41be3c-host\") pod \"node-ca-6htsx\" (UID: \"0a0c3e91-a0ba-4045-a210-240fbc41be3c\") " pod="openshift-image-registry/node-ca-6htsx"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031637    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031661    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031667    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031671    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-multus\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031680    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-etc-selinux\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031696    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031712    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.031720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031727    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031746    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031751    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-multus-certs\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031769    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031776    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-var-lib-cni-bin\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031785    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-slash\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031775    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fd84fe1e-ce28-4634-b84c-fa50c5dc2be2-hosts-file\") pod \"node-resolver-dpmg7\" (UID: \"fd84fe1e-ce28-4634-b84c-fa50c5dc2be2\") " pod="openshift-dns/node-resolver-dpmg7"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031798    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36293e5-2420-4631-ad8b-5747f97896f1-root\") pod \"node-exporter-wrh8c\" (UID: \"a36293e5-2420-4631-ad8b-5747f97896f1\") " pod="openshift-monitoring/node-exporter-wrh8c"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031809    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/200522d1-e408-41f4-be88-15c3600c7225-crio-socket\") pod \"insights-runtime-extractor-gk9d8\" (UID: \"200522d1-e408-41f4-be88-15c3600c7225\") " pod="openshift-insights/insights-runtime-extractor-gk9d8"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031809    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031834    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031840    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-conf-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031858    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031881    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4616fa40-654c-46fd-9f47-aeada7820c79-host-slash\") pod \"iptables-alerter-l4fvp\" (UID: \"4616fa40-654c-46fd-9f47-aeada7820c79\") " pod="openshift-network-operator/iptables-alerter-l4fvp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031881    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031916    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031924    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-run\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031893    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-var-lib-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031939    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.031976    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3ece716d-9af9-434f-85bf-9e6e6fb988f3-kubelet-config\") pod \"global-pull-secret-syncer-8tx9f\" (UID: \"3ece716d-9af9-434f-85bf-9e6e6fb988f3\") " pod="kube-system/global-pull-secret-syncer-8tx9f"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032001    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032018    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-lib-modules\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032025    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032043    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032048    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-var-lib-kubelet\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032057    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032078    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032081    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-multus-cni-dir\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032104    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032109    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-etc-openvswitch\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032121    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-systemd-units\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032135    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032148    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-socket-dir\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032155    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032109    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-log-socket\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032178    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-kubelet\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032182    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032200    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032211    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032233    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032254    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032275    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032293    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-system-cni-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032297    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-sysctl-conf\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032300    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032307    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c0e95d56-77bd-4a2d-b839-326a87300313-sys-fs\") pod \"aws-ebs-csi-driver-node-kw4rp\" (UID: \"c0e95d56-77bd-4a2d-b839-326a87300313\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kw4rp"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032315    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dab969bc-1102-4323-816b-eb8944cc4d82-host-run-netns\") pod \"multus-4gqzk\" (UID: \"dab969bc-1102-4323-816b-eb8944cc4d82\") " pod="openshift-multus/multus-4gqzk"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-host-run-netns\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032331    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032328    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-cnibin\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032350    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032366    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032397    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032421    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72-run-systemd\") pod \"ovnkube-node-b7h7z\" (UID: \"2e8d5a65-1ac4-4581-bcc1-0dbcf11d2a72\") " pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032425    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-systemd\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032449    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/47b52073-13fa-4418-b8fd-f9866952b917-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7jh9h\" (UID: \"47b52073-13fa-4418-b8fd-f9866952b917\") " pod="openshift-multus/multus-additional-cni-plugins-7jh9h"
Nov 03 11:51:35.032636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.032452    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/488fc93c-6c34-49ae-83b9-9e6f930ad2b7-etc-modprobe-d\") pod \"tuned-m2svl\" (UID: \"488fc93c-6c34-49ae-83b9-9e6f930ad2b7\") " pod="openshift-cluster-node-tuning-operator/tuned-m2svl"
Nov 03 11:51:35.152230 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.152143    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:51:35.850089 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.850055    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:35.854570 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.854544    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:35.975572 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:35.975543    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:36.011334 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.011308    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhqfb"
Nov 03 11:51:36.142003 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.141976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:36.309490 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.309459    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:36.514611 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.514507    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:36.648821 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.648790    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:36.654633 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:36.654609    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7748596f65-48dhl"
Nov 03 11:51:37.176656 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:37.176620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:37.937308 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:37.937273    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:39.150440 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:39.150406    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:51:39.546674 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:39.546590    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:39.714438 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:39.714402    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:40.195905 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:40.195872    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:40.366040 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:40.366009    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:40.775503 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:40.775473    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:40.964024 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:40.963994    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:42.003926 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:42.003895    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:42.445096 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:42.445069    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:42.778899 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:42.778814    5784 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 11:51:43.167596 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:43.167559    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:43.327720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:43.327684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:43.806161 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:43.806132    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:51:44.001943 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:44.001911    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:44.166790 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:44.166758    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:44.740422 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:44.740358    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:44.887642 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:44.887608    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:47.942176 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:47.942146    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:51:48.764461 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:48.764427    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:50.370806 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:50.370774    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:51:51.876004 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:51.875974    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-785td"
Nov 03 11:51:52.371322 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:52.371228    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:52.522137 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:52.522103    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:54.677748 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:54.677714    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:54.834896 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:54.834864    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:51:56.131667 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:56.131640    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b7h7z"
Nov 03 11:51:56.470509 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:51:56.470434    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:51:59.823799 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:51:59.823742    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5db5589b4-9v2nm" podUID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" containerName="console" containerID="cri-o://f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157" gracePeriod=15
Nov 03 11:52:00.104999 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.104932    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:52:00.248136 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248102    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248157    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248187    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248223    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248241    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248259    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248319 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248292    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248654 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248324    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") pod \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\" (UID: \"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a\") "
Nov 03 11:52:00.248654 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248464    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:00.248801 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248774    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:00.248863 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248800    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:00.248902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.248868    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:00.249215 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.249165    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:00.250899 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.250871    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:00.251159 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.251136    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:00.251406 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.251368    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq" pod "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" (UID: "65c1d3e1-850e-4d74-be2b-d9f1ce17d40a"). InnerVolumeSpecName "kube-api-access-49wkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:00.349866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349830    5784 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-oauth-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.349866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349859    5784 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-tmp\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.349866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349870    5784 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.350095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349881    5784 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.350095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349890    5784 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.350095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349899    5784 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-console-oauth-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.350095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349907    5784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-trusted-ca-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.350095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.349916    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49wkq\" (UniqueName: \"kubernetes.io/projected/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a-kube-api-access-49wkq\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:00.770450 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.770392    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-86dfb58d96-wn4hk" podUID="b543b127-aa97-4f98-863e-ca16072e4031" containerName="console" containerID="cri-o://86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8" gracePeriod=15
Nov 03 11:52:00.848345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.848313    5784 generic.go:356] "Generic (PLEG): container finished" podID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" containerID="f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157" exitCode=0
Nov 03 11:52:00.848771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.848360    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db5589b4-9v2nm" event={"ID":"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a","Type":"ContainerDied","Data":"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"}
Nov 03 11:52:00.848771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.848388    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db5589b4-9v2nm"
Nov 03 11:52:00.848771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.848412    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db5589b4-9v2nm" event={"ID":"65c1d3e1-850e-4d74-be2b-d9f1ce17d40a","Type":"ContainerDied","Data":"96c8c862425e781a9ace46c0fc76190d460e01c7811ecb53674c04030692b607"}
Nov 03 11:52:00.848771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.848444    5784 scope.go:122] "RemoveContainer" containerID="f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"
Nov 03 11:52:00.880966 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.880942    5784 scope.go:122] "RemoveContainer" containerID="f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"
Nov 03 11:52:00.881273 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:00.881254    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157\": container with ID starting with f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157 not found: ID does not exist" containerID="f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"
Nov 03 11:52:00.881346 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.881280    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157"} err="failed to get container status \"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157\": rpc error: code = NotFound desc = could not find container \"f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157\": container with ID starting with f2e8f43145772925f9e0836dce658dc92a8419b29acd290bf95533faacbfd157 not found: ID does not exist"
Nov 03 11:52:00.998432 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:00.998402    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:52:01.044486 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.044461    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5db5589b4-9v2nm"]
Nov 03 11:52:01.077960 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.077937    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:52:01.155364 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155334    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155491 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155415    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155491 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155447    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155491 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155467    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155491 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155486    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155680 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155513    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155680 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155548    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") pod \"b543b127-aa97-4f98-863e-ca16072e4031\" (UID: \"b543b127-aa97-4f98-863e-ca16072e4031\") "
Nov 03 11:52:01.155680 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155665    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:01.155823 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155791    5784 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b543b127-aa97-4f98-863e-ca16072e4031-tmp\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.155937 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.155911    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:01.156098 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.156015    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:01.156098 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.156052    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:01.157837 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.157808    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:01.157837 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.157807    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "kube-api-access-jjdv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:01.157969 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.157884    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config" pod "b543b127-aa97-4f98-863e-ca16072e4031" (UID: "b543b127-aa97-4f98-863e-ca16072e4031"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:01.257142 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257106    5784 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-oauth-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.257142 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257137    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjdv9\" (UniqueName: \"kubernetes.io/projected/b543b127-aa97-4f98-863e-ca16072e4031-kube-api-access-jjdv9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.257142 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257147    5784 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-console-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.257371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257156    5784 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-oauth-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.257371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257166    5784 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b543b127-aa97-4f98-863e-ca16072e4031-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.257371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.257174    5784 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b543b127-aa97-4f98-863e-ca16072e4031-console-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:01.527534 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.527496    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:01.672254 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.672219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:01.679106 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.679079    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" path="/var/lib/kubelet/pods/65c1d3e1-850e-4d74-be2b-d9f1ce17d40a/volumes"
Nov 03 11:52:01.853946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.853860    5784 generic.go:356] "Generic (PLEG): container finished" podID="b543b127-aa97-4f98-863e-ca16072e4031" containerID="86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8" exitCode=0
Nov 03 11:52:01.853946 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.853914    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86dfb58d96-wn4hk"
Nov 03 11:52:01.854477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.853950    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dfb58d96-wn4hk" event={"ID":"b543b127-aa97-4f98-863e-ca16072e4031","Type":"ContainerDied","Data":"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"}
Nov 03 11:52:01.854477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.853999    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86dfb58d96-wn4hk" event={"ID":"b543b127-aa97-4f98-863e-ca16072e4031","Type":"ContainerDied","Data":"b56a7c86bc6594d7a4e4b2368f42c2874f3a68f15952fab3142f4e2d29018a51"}
Nov 03 11:52:01.854477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.854027    5784 scope.go:122] "RemoveContainer" containerID="86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"
Nov 03 11:52:01.861723 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.861674    5784 scope.go:122] "RemoveContainer" containerID="86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"
Nov 03 11:52:01.861943 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:01.861922    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8\": container with ID starting with 86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8 not found: ID does not exist" containerID="86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"
Nov 03 11:52:01.861985 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:01.861957    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8"} err="failed to get container status \"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8\": rpc error: code = NotFound desc = could not find container \"86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8\": container with ID starting with 86bf52e3a4a460683b3c649ddd4acd0e6d0ccf97866632cd8ef22c31b21bdac8 not found: ID does not exist"
Nov 03 11:52:02.005356 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:02.005327    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:52:02.051326 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:02.051291    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86dfb58d96-wn4hk"]
Nov 03 11:52:03.678466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:03.678435    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b543b127-aa97-4f98-863e-ca16072e4031" path="/var/lib/kubelet/pods/b543b127-aa97-4f98-863e-ca16072e4031/volumes"
Nov 03 11:52:03.808583 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:03.808554    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-58msl"
Nov 03 11:52:09.672797 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:09.672755    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:09.830556 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:09.830524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:10.375584 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:10.375553    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:52:10.379210 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:10.379189    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-96685b96b-5s9hm"
Nov 03 11:52:16.151557 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:16.151527    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:16.304016 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:16.303981    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:18.226634 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.226594    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:18.227123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.226972    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="alertmanager" containerID="cri-o://8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772" gracePeriod=120
Nov 03 11:52:18.227123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.227025    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-metric" containerID="cri-o://e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289" gracePeriod=120
Nov 03 11:52:18.227123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.227042    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy" containerID="cri-o://5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069" gracePeriod=120
Nov 03 11:52:18.227123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.227065    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="prom-label-proxy" containerID="cri-o://877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b" gracePeriod=120
Nov 03 11:52:18.227363 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.227097    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="config-reloader" containerID="cri-o://f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a" gracePeriod=120
Nov 03 11:52:18.227363 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.227039    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-web" containerID="cri-o://c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd" gracePeriod=120
Nov 03 11:52:18.918955 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918918    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b" exitCode=0
Nov 03 11:52:18.918955 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918947    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289" exitCode=0
Nov 03 11:52:18.918955 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918955    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069" exitCode=0
Nov 03 11:52:18.918955 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918962    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a" exitCode=0
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918969    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772" exitCode=0
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.918989    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"}
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.919053    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"}
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.919079    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"}
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.919104    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"}
Nov 03 11:52:18.919229 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:18.919128    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"}
Nov 03 11:52:19.473114 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.473090    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:19.610036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.609956    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.609994    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610020    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610255 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610040    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610255 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610229    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610278    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610316    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610463 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610345    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610463 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610432    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610462    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610510    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610536    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:19.610569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610550    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610569 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610557    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:19.610829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610582    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") pod \"1c41b69d-ee39-45cb-8a06-66b5029f2952\" (UID: \"1c41b69d-ee39-45cb-8a06-66b5029f2952\") "
Nov 03 11:52:19.610829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610803    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-metrics-client-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.610829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.610825    5784 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-main-db\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.611365 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.611335    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:52:19.613579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.613540    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.613963 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.613598    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "kube-api-access-4p2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:19.613963 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.613663    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:52:19.613963 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.613757    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.613963 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.613933    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.614188 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.614106    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:52:19.614188 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.614133    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.614409 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.614370    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.617371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.617351    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.624270 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.624241    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config" pod "1c41b69d-ee39-45cb-8a06-66b5029f2952" (UID: "1c41b69d-ee39-45cb-8a06-66b5029f2952"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:52:19.711856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711831    5784 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-cluster-tls-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.711856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711856    5784 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-web-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711867    5784 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711876    5784 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711885    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2lk\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-kube-api-access-4p2lk\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711896    5784 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711905    5784 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1c41b69d-ee39-45cb-8a06-66b5029f2952-tls-assets\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711914    5784 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711924    5784 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1c41b69d-ee39-45cb-8a06-66b5029f2952-config-out\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711933    5784 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1c41b69d-ee39-45cb-8a06-66b5029f2952-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.712022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.711942    5784 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1c41b69d-ee39-45cb-8a06-66b5029f2952-secret-alertmanager-main-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:52:19.925550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.925516    5784 generic.go:356] "Generic (PLEG): container finished" podID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerID="c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd" exitCode=0
Nov 03 11:52:19.925707 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.925596    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:19.925707 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.925606    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"}
Nov 03 11:52:19.925707 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.925678    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1c41b69d-ee39-45cb-8a06-66b5029f2952","Type":"ContainerDied","Data":"666137b783eeee3538859db4a720810d09e654b33c122bf1be5ff917924cccb8"}
Nov 03 11:52:19.925821 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.925716    5784 scope.go:122] "RemoveContainer" containerID="877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"
Nov 03 11:52:19.933126 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.933099    5784 scope.go:122] "RemoveContainer" containerID="e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"
Nov 03 11:52:19.939580 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.939563    5784 scope.go:122] "RemoveContainer" containerID="5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"
Nov 03 11:52:19.945843 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.945825    5784 scope.go:122] "RemoveContainer" containerID="c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"
Nov 03 11:52:19.954061 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.954043    5784 scope.go:122] "RemoveContainer" containerID="f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"
Nov 03 11:52:19.960624 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.960607    5784 scope.go:122] "RemoveContainer" containerID="8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"
Nov 03 11:52:19.967222 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.967204    5784 scope.go:122] "RemoveContainer" containerID="44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2"
Nov 03 11:52:19.973661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.973645    5784 scope.go:122] "RemoveContainer" containerID="877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"
Nov 03 11:52:19.973910 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.973889    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b\": container with ID starting with 877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b not found: ID does not exist" containerID="877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"
Nov 03 11:52:19.973967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.973929    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b"} err="failed to get container status \"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b\": rpc error: code = NotFound desc = could not find container \"877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b\": container with ID starting with 877ba822f3f3a859b9fa1b0ba23dca981689b11fe653e8988ef75c5658f4390b not found: ID does not exist"
Nov 03 11:52:19.973967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.973953    5784 scope.go:122] "RemoveContainer" containerID="e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"
Nov 03 11:52:19.974212 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.974191    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289\": container with ID starting with e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289 not found: ID does not exist" containerID="e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"
Nov 03 11:52:19.974270 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974218    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289"} err="failed to get container status \"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289\": rpc error: code = NotFound desc = could not find container \"e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289\": container with ID starting with e451e2afd4c9f0a534a3ee0cc5c22be95e095f873ec7a0dbc034d0ca9025c289 not found: ID does not exist"
Nov 03 11:52:19.974270 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974236    5784 scope.go:122] "RemoveContainer" containerID="5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"
Nov 03 11:52:19.974565 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.974549    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069\": container with ID starting with 5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069 not found: ID does not exist" containerID="5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"
Nov 03 11:52:19.974626 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974569    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069"} err="failed to get container status \"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069\": rpc error: code = NotFound desc = could not find container \"5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069\": container with ID starting with 5590f918a4acaee4e8c51e658b34a991fdb9ea2c2a02d4fc9b35055402c34069 not found: ID does not exist"
Nov 03 11:52:19.974626 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974583    5784 scope.go:122] "RemoveContainer" containerID="c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"
Nov 03 11:52:19.974842 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.974825    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd\": container with ID starting with c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd not found: ID does not exist" containerID="c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"
Nov 03 11:52:19.974901 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974851    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd"} err="failed to get container status \"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd\": rpc error: code = NotFound desc = could not find container \"c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd\": container with ID starting with c79fce29f0e3a1d248762f69adea81c6ef90d0e2fef7718bc1c6205e84a3eefd not found: ID does not exist"
Nov 03 11:52:19.974901 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.974872    5784 scope.go:122] "RemoveContainer" containerID="f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"
Nov 03 11:52:19.975096 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.975079    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a\": container with ID starting with f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a not found: ID does not exist" containerID="f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"
Nov 03 11:52:19.975133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.975101    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a"} err="failed to get container status \"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a\": rpc error: code = NotFound desc = could not find container \"f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a\": container with ID starting with f7d4ed78c07357d6692ea6eccad02d8738ce545a15aa779478beb5c400303e8a not found: ID does not exist"
Nov 03 11:52:19.975133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.975114    5784 scope.go:122] "RemoveContainer" containerID="8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"
Nov 03 11:52:19.975320 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.975303    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772\": container with ID starting with 8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772 not found: ID does not exist" containerID="8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"
Nov 03 11:52:19.975412 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.975327    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772"} err="failed to get container status \"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772\": rpc error: code = NotFound desc = could not find container \"8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772\": container with ID starting with 8f0ab89d5842336d5168ca484de5469bbe356720c5157baf35125a63b95bb772 not found: ID does not exist"
Nov 03 11:52:19.975412 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.975344    5784 scope.go:122] "RemoveContainer" containerID="44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2"
Nov 03 11:52:19.975586 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:19.975570    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2\": container with ID starting with 44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2 not found: ID does not exist" containerID="44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2"
Nov 03 11:52:19.975619 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:19.975590    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2"} err="failed to get container status \"44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2\": rpc error: code = NotFound desc = could not find container \"44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2\": container with ID starting with 44190ac90b8598219a905113271cef5d6c8a3887f7ada220a31493aa9a0878c2 not found: ID does not exist"
Nov 03 11:52:20.079340 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.079310    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:20.125363 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.125325    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:20.125825 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.125781    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:20.126145 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126125    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b543b127-aa97-4f98-863e-ca16072e4031" containerName="console"
Nov 03 11:52:20.126270 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126254    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b543b127-aa97-4f98-863e-ca16072e4031" containerName="console"
Nov 03 11:52:20.126270 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126271    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126278    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126288    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-metric"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126293    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-metric"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126300    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="alertmanager"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126305    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="alertmanager"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126322    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="init-config-reloader"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126327    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="init-config-reloader"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126335    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="prom-label-proxy"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126340    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="prom-label-proxy"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126346    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="config-reloader"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126351    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="config-reloader"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126359    5784 cpu_manager.go:414] "containerMap: removing container" podUID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" containerName="console"
Nov 03 11:52:20.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126364    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" containerName="console"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126409    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-web"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126415    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-web"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126471    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b543b127-aa97-4f98-863e-ca16072e4031" containerName="console"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126477    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126484    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-metric"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126488    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="alertmanager"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126495    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="init-config-reloader"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126501    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="prom-label-proxy"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126505    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="config-reloader"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126510    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c1d3e1-850e-4d74-be2b-d9f1ce17d40a" containerName="console"
Nov 03 11:52:20.126749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.126515    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" containerName="kube-rbac-proxy-web"
Nov 03 11:52:20.129817 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.129802    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179211    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179218    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179218    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179249    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179279    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 11:52:20.179335 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179287    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 11:52:20.179662 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179428    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 11:52:20.179662 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179432    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 11:52:20.179662 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.179468    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-46zdg\""
Nov 03 11:52:20.225231 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.225204    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 11:52:20.236055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.236025    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:20.316030 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316000    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316206 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316040    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316206 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316064    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316206 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316081    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316206 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316196    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316330 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316254    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316330 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316283    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316330 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316304    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316450 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316417    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4k7\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-kube-api-access-ps4k7\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316450 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316441    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-web-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316517 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316460    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-out\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316547 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316527    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.316579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.316553    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.416978 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.416942    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.416978 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.416984    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417011    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417045    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417243 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417209    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417287 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417247    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417322 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417294    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417368 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417326    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417368 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417352    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417497 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417419    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4k7\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-kube-api-access-ps4k7\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417497 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417443    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417497 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417457    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-web-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417497 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417479    5784 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 11:52:20.417694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417503    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-out\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.417694 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.417540    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.418334 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.418003    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.418334 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.418263    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b299cb60-8c1f-4a01-befa-fece4d244bbb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420422 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420401    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420698 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420655    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420698 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420676    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420867 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420807    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420922 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420864    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.420922 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.420893    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.421154 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.421134    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-web-config\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.421215 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.421195    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b299cb60-8c1f-4a01-befa-fece4d244bbb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.421871 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.421856    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b299cb60-8c1f-4a01-befa-fece4d244bbb-config-out\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.471435 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.471334    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4k7\" (UniqueName: \"kubernetes.io/projected/b299cb60-8c1f-4a01-befa-fece4d244bbb-kube-api-access-ps4k7\") pod \"alertmanager-main-0\" (UID: \"b299cb60-8c1f-4a01-befa-fece4d244bbb\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:20.746852 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:20.746784    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 11:52:21.207912 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:21.207878    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 11:52:21.239093 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:52:21.239062    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb299cb60_8c1f_4a01_befa_fece4d244bbb.slice/crio-70b7c6b611f56b78ba3f2fb5ae81c8c29083462da2c278dc0b634dd3d19c2776 WatchSource:0}: Error finding container 70b7c6b611f56b78ba3f2fb5ae81c8c29083462da2c278dc0b634dd3d19c2776: Status 404 returned error can't find the container with id 70b7c6b611f56b78ba3f2fb5ae81c8c29083462da2c278dc0b634dd3d19c2776
Nov 03 11:52:21.679476 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:21.679445    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c41b69d-ee39-45cb-8a06-66b5029f2952" path="/var/lib/kubelet/pods/1c41b69d-ee39-45cb-8a06-66b5029f2952/volumes"
Nov 03 11:52:21.935609 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:21.935523    5784 generic.go:356] "Generic (PLEG): container finished" podID="b299cb60-8c1f-4a01-befa-fece4d244bbb" containerID="4b583e5aa3591b2bbe2f5e294d5cff3ec7064a6c64e2a266c4789f9f4878a22d" exitCode=0
Nov 03 11:52:21.935609 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:21.935589    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerDied","Data":"4b583e5aa3591b2bbe2f5e294d5cff3ec7064a6c64e2a266c4789f9f4878a22d"}
Nov 03 11:52:21.936116 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:21.935637    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"70b7c6b611f56b78ba3f2fb5ae81c8c29083462da2c278dc0b634dd3d19c2776"}
Nov 03 11:52:22.040214 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.040185    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5895657d88-9n7mx"]
Nov 03 11:52:22.042795 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.042775    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.092668 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092640    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 11:52:22.092807 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092759    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 11:52:22.092871 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092640    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 11:52:22.092871 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092758    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 11:52:22.092970 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092869    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 11:52:22.092970 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.092937    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn5vg\""
Nov 03 11:52:22.139266 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.139246    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 11:52:22.182301 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.182275    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5895657d88-9n7mx"]
Nov 03 11:52:22.232563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232540    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232575    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-federate-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232595    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232614    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232790 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232756    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnl47\" (UniqueName: \"kubernetes.io/projected/b77b9219-acad-4786-91ad-6bfe81c64ffc-kube-api-access-pnl47\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232838 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232795    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-metrics-client-ca\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232838 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232814    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.232903 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.232869    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-serving-certs-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.333843 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333813    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333853    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-federate-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333871    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333888    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333911    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnl47\" (UniqueName: \"kubernetes.io/projected/b77b9219-acad-4786-91ad-6bfe81c64ffc-kube-api-access-pnl47\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333930    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-metrics-client-ca\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333951    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.333986    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-serving-certs-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334899 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.334867    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-metrics-client-ca\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.334899 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.334893    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-serving-certs-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.335184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.335160    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.337055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.337033    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-telemeter-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.337154 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.337039    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-federate-client-tls\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.337154 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.337073    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.337154 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.337083    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b77b9219-acad-4786-91ad-6bfe81c64ffc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.386889 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.386854    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnl47\" (UniqueName: \"kubernetes.io/projected/b77b9219-acad-4786-91ad-6bfe81c64ffc-kube-api-access-pnl47\") pod \"telemeter-client-5895657d88-9n7mx\" (UID: \"b77b9219-acad-4786-91ad-6bfe81c64ffc\") " pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.660876 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.660844    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx"
Nov 03 11:52:22.942555 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942482    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"6d4036a8aeceb4c593014564689dcd190225c11f5bd1692c2b2dbea2f0486870"}
Nov 03 11:52:22.942555 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942546    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"fe15bb1a3f16955aff0ea0c6296ca5b4330b1bd7b0938c905bf9cb7a5d9676da"}
Nov 03 11:52:22.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942571    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"56746f2e9d69e49bf2c15f43df6c8ceeaf87c19e5d320e199a5f41e534ccfe0b"}
Nov 03 11:52:22.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942597    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"766de5a21dfd893890bac4e40c51a3858b913fb037ba071cf204627496e4de5f"}
Nov 03 11:52:22.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942619    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"dc25c07f727c3f8d06d28495705da570dd95edcecdbfb6e401bf836b24966ad1"}
Nov 03 11:52:22.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:22.942641    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b299cb60-8c1f-4a01-befa-fece4d244bbb","Type":"ContainerStarted","Data":"e23b6f545fc645635bbad525e5d393ca9c0a79aae3b6804787b0e79122589860"}
Nov 03 11:52:23.058787 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:23.058732    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.045942774 podStartE2EDuration="3.058712231s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.769457ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:52:23.057335936 +0000 UTC m=+49.499538334" watchObservedRunningTime="2026-11-03 11:52:23.058712231 +0000 UTC m=+49.500914549"
Nov 03 11:52:23.115985 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:23.115509    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5895657d88-9n7mx"]
Nov 03 11:52:23.118537 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:52:23.118500    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77b9219_acad_4786_91ad_6bfe81c64ffc.slice/crio-7bcbabb289607e07fc94c72ee495a4a3270ff01c0546b6710ddbe4515bcc56a4 WatchSource:0}: Error finding container 7bcbabb289607e07fc94c72ee495a4a3270ff01c0546b6710ddbe4515bcc56a4: Status 404 returned error can't find the container with id 7bcbabb289607e07fc94c72ee495a4a3270ff01c0546b6710ddbe4515bcc56a4
Nov 03 11:52:23.120784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:23.120761    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:52:23.949370 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:23.949336    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx" event={"ID":"b77b9219-acad-4786-91ad-6bfe81c64ffc","Type":"ContainerStarted","Data":"7bcbabb289607e07fc94c72ee495a4a3270ff01c0546b6710ddbe4515bcc56a4"}
Nov 03 11:52:25.205502 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:25.205479    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:25.959320 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:25.959281    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx" event={"ID":"b77b9219-acad-4786-91ad-6bfe81c64ffc","Type":"ContainerStarted","Data":"c8cb2fa1f83b0a255278f1038fff27fb3f5d8a9984c12b40b127ac4c8261a9b7"}
Nov 03 11:52:25.959524 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:25.959363    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx" event={"ID":"b77b9219-acad-4786-91ad-6bfe81c64ffc","Type":"ContainerStarted","Data":"9c420cbb0c3858583e1265edef58136e01192b6c9561baa9c4f35596373530e8"}
Nov 03 11:52:25.959524 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:25.959394    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx" event={"ID":"b77b9219-acad-4786-91ad-6bfe81c64ffc","Type":"ContainerStarted","Data":"df20ef5c6942d3613e9fbdae8d3249c515b83cb1b5eb695b23e55a5b8b44a5ab"}
Nov 03 11:52:26.176030 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.175977    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5895657d88-9n7mx" podStartSLOduration=3.240262395 podStartE2EDuration="5.175961741s" totalImagesPullingTime="1.935699346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:52:26.175223134 +0000 UTC m=+52.617425439" watchObservedRunningTime="2026-11-03 11:52:26.175961741 +0000 UTC m=+52.618164047"
Nov 03 11:52:26.470631 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:26.470595    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:52:26.583473 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.583440    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:52:26.585680 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.585664    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.640849 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.640808    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:52:26.674544 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674513    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674544 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674546    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674745 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674568    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674745 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674677    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674745 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674703    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674745 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674744    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4jzd\" (UniqueName: \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674884 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674813    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.674884 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.674850    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.775874 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775797    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.775874 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775831    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.775874 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775853    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775882    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775905    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775968    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4jzd\" (UniqueName: \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.775992    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776033    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776588 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776562    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776716 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776704    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776918 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776727    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.776918 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776818    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.777003 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.776985    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.778433 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.778412    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.778638 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.778620    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.829733 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.829709    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4jzd\" (UniqueName: \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\") pod \"console-79d4798df4-pzlw5\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") " pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:26.879632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.879599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:26.895460 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:26.895436    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:27.256935 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:27.256904    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:52:27.261157 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:52:27.261129    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec3fa67_9a08_46d3_8274_cdf144a62167.slice/crio-a501f3ce1a525084ac91b6b1e4a44cc689d035b99d868799f6bc2ea2ca4d067a WatchSource:0}: Error finding container a501f3ce1a525084ac91b6b1e4a44cc689d035b99d868799f6bc2ea2ca4d067a: Status 404 returned error can't find the container with id a501f3ce1a525084ac91b6b1e4a44cc689d035b99d868799f6bc2ea2ca4d067a
Nov 03 11:52:27.264624 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:27.264597    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:27.419521 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:27.419482    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:27.968309 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:27.968275    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4798df4-pzlw5" event={"ID":"3ec3fa67-9a08-46d3-8274-cdf144a62167","Type":"ContainerStarted","Data":"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"}
Nov 03 11:52:27.968835 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:27.968332    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4798df4-pzlw5" event={"ID":"3ec3fa67-9a08-46d3-8274-cdf144a62167","Type":"ContainerStarted","Data":"a501f3ce1a525084ac91b6b1e4a44cc689d035b99d868799f6bc2ea2ca4d067a"}
Nov 03 11:52:28.075689 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:28.075631    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-79d4798df4-pzlw5" podStartSLOduration=2.075616801 podStartE2EDuration="2.075616801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:52:28.075369079 +0000 UTC m=+54.517571384" watchObservedRunningTime="2026-11-03 11:52:28.075616801 +0000 UTC m=+54.517819100"
Nov 03 11:52:36.896371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:36.896316    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:36.896371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:36.896405    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:36.901082 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:36.901061    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:37.004639 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:37.004608    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:52:37.142110 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:37.142072    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:52:41.091796 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:52:41.091761    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:52:56.470158 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:52:56.470111    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:53:02.108477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.108446    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:02.161911 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.161871    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-bbf444864-cm8fl" podUID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" containerName="console" containerID="cri-o://dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e" gracePeriod=15
Nov 03 11:53:02.407520 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.407497    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:53:02.581696 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581663    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.581861 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581724    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.581861 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581751    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.581861 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581790    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.581861 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581828    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.582055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581863    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.582055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581916    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.582055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.581958    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") pod \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\" (UID: \"3d4550a1-3f78-4685-aa82-2546c2aa7b41\") "
Nov 03 11:53:02.582201 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582068    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:53:02.582255 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582099    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:53:02.582311 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582279    5784 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-oauth-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.582311 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582301    5784 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4550a1-3f78-4685-aa82-2546c2aa7b41-tmp\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.582449 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582346    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:53:02.582512 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582473    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:53:02.582597 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.582563    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:53:02.584107 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.584067    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:53:02.584208 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.584186    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "kube-api-access-w7qmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:53:02.584317 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.584298    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config" pod "3d4550a1-3f78-4685-aa82-2546c2aa7b41" (UID: "3d4550a1-3f78-4685-aa82-2546c2aa7b41"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683073    5784 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683103    5784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-trusted-ca-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683113    5784 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683122    5784 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d4550a1-3f78-4685-aa82-2546c2aa7b41-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683132    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qmr\" (UniqueName: \"kubernetes.io/projected/3d4550a1-3f78-4685-aa82-2546c2aa7b41-kube-api-access-w7qmr\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:02.683158 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:02.683142    5784 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d4550a1-3f78-4685-aa82-2546c2aa7b41-console-oauth-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:53:03.098543 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.098452    5784 generic.go:356] "Generic (PLEG): container finished" podID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" containerID="dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e" exitCode=0
Nov 03 11:53:03.098543 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.098507    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bbf444864-cm8fl"
Nov 03 11:53:03.098543 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.098533    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf444864-cm8fl" event={"ID":"3d4550a1-3f78-4685-aa82-2546c2aa7b41","Type":"ContainerDied","Data":"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"}
Nov 03 11:53:03.098774 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.098606    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bbf444864-cm8fl" event={"ID":"3d4550a1-3f78-4685-aa82-2546c2aa7b41","Type":"ContainerDied","Data":"27aec2e03b371cfc540ac49b3e2eb09aa4dacb1b279086a360ed217f3321116d"}
Nov 03 11:53:03.098774 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.098627    5784 scope.go:122] "RemoveContainer" containerID="dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"
Nov 03 11:53:03.106704 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.106684    5784 scope.go:122] "RemoveContainer" containerID="dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"
Nov 03 11:53:03.106983 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:53:03.106966    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e\": container with ID starting with dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e not found: ID does not exist" containerID="dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"
Nov 03 11:53:03.107029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.106991    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e"} err="failed to get container status \"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e\": rpc error: code = NotFound desc = could not find container \"dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e\": container with ID starting with dc0dd55e05e4bc983c8a27addbd8eb39462985c45d0173e5a608a25a0756329e not found: ID does not exist"
Nov 03 11:53:03.249604 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.249573    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:53:03.295492 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.295435    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bbf444864-cm8fl"]
Nov 03 11:53:03.681210 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:03.681174    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" path="/var/lib/kubelet/pods/3d4550a1-3f78-4685-aa82-2546c2aa7b41/volumes"
Nov 03 11:53:26.470812 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:53:26.470729    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:53:33.794071 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:33.794032    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:37.829696 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.829665    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f4876bd-t6mwn"]
Nov 03 11:53:37.830127 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.830028    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" containerName="console"
Nov 03 11:53:37.830127 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.830042    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" containerName="console"
Nov 03 11:53:37.830196 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.830129    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d4550a1-3f78-4685-aa82-2546c2aa7b41" containerName="console"
Nov 03 11:53:37.834334 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.834316    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.887368 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.887336    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4876bd-t6mwn"]
Nov 03 11:53:37.963465 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963434    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-trusted-ca-bundle\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963641 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963473    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-oauth-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963641 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963498    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-service-ca\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963641 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963609    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-oauth-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963760 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963694    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4dmr\" (UniqueName: \"kubernetes.io/projected/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-kube-api-access-c4dmr\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963760 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963715    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963760 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963736    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-tmp\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:37.963882 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:37.963865    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064773 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064739    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064782    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-trusted-ca-bundle\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064802    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-oauth-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064822    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-service-ca\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064839    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-oauth-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064859    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4dmr\" (UniqueName: \"kubernetes.io/projected/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-kube-api-access-c4dmr\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064883    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.064933 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.064909    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-tmp\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.065677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.065652    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-oauth-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.065839 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.065809    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-service-ca\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.065956 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.065714    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.065956 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.065683    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-tmp\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.066121 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.066104    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-trusted-ca-bundle\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.067351 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.067326    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-oauth-config\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.067515 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.067494    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-console-serving-cert\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.118037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.117965    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4dmr\" (UniqueName: \"kubernetes.io/projected/fb10eec3-2578-4f1a-8c1d-20bbb50d6c15-kube-api-access-c4dmr\") pod \"console-7f4876bd-t6mwn\" (UID: \"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15\") " pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.133581 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.133552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:38.145463 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.145439    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:38.289400 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.289336    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:38.506322 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.506290    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f4876bd-t6mwn"]
Nov 03 11:53:38.509457 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:53:38.509430    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb10eec3_2578_4f1a_8c1d_20bbb50d6c15.slice/crio-95976cea6a1b2225e9353f1e2be2f581d2b89ae048eb79b3b6d326196df5e869 WatchSource:0}: Error finding container 95976cea6a1b2225e9353f1e2be2f581d2b89ae048eb79b3b6d326196df5e869: Status 404 returned error can't find the container with id 95976cea6a1b2225e9353f1e2be2f581d2b89ae048eb79b3b6d326196df5e869
Nov 03 11:53:38.517204 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.517184    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:38.678991 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:38.678962    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:39.228371 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:39.228332    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4876bd-t6mwn" event={"ID":"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15","Type":"ContainerStarted","Data":"05aa7f94491b75b0d47ec981a1458828db3ae79e6481e8e9880a49e0fd829a73"}
Nov 03 11:53:39.228825 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:39.228408    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f4876bd-t6mwn" event={"ID":"fb10eec3-2578-4f1a-8c1d-20bbb50d6c15","Type":"ContainerStarted","Data":"95976cea6a1b2225e9353f1e2be2f581d2b89ae048eb79b3b6d326196df5e869"}
Nov 03 11:53:39.336309 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:39.336260    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f4876bd-t6mwn" podStartSLOduration=2.33624446 podStartE2EDuration="2.33624446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:53:39.335599459 +0000 UTC m=+125.777801778" watchObservedRunningTime="2026-11-03 11:53:39.33624446 +0000 UTC m=+125.778446764"
Nov 03 11:53:41.404133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:41.404105    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:46.530745 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:46.530713    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:46.694064 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:46.694028    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:48.146491 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:48.146448    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:48.146924 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:48.146512    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:48.151030 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:48.151008    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:48.269392 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:48.269356    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f4876bd-t6mwn"
Nov 03 11:53:48.403809 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:48.403724    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:53:54.673330 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:54.673293    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:54.826153 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:53:54.826119    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:53:56.470515 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:53:56.470483    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:54:13.422674 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.422635    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-79d4798df4-pzlw5" podUID="3ec3fa67-9a08-46d3-8274-cdf144a62167" containerName="console" containerID="cri-o://2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2" gracePeriod=15
Nov 03 11:54:13.663045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.663018    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:54:13.664466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664444    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664475    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664500    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664517    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\" (UniqueName: \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664747 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664712    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664752    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664859 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664809    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664915 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664863    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\") pod \"3ec3fa67-9a08-46d3-8274-cdf144a62167\" (UID: \"3ec3fa67-9a08-46d3-8274-cdf144a62167\") "
Nov 03 11:54:13.664915 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664873    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:54:13.664915 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.664897    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:54:13.665169 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.665110    5784 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-oauth-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.665169 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.665135    5784 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.665299 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.665268    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:54:13.665299 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.665273    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:54:13.665299 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.665283    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 11:54:13.666584 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.666564    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "kube-api-access-t4jzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:54:13.667105 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.667085    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:54:13.667156 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.667105    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert" pod "3ec3fa67-9a08-46d3-8274-cdf144a62167" (UID: "3ec3fa67-9a08-46d3-8274-cdf144a62167"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:54:13.765714 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765673    5784 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-oauth-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.765714 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765711    5784 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-config\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.765714 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765725    5784 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ec3fa67-9a08-46d3-8274-cdf144a62167-trusted-ca-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.765998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765740    5784 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ec3fa67-9a08-46d3-8274-cdf144a62167-tmp\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.765998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765754    5784 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec3fa67-9a08-46d3-8274-cdf144a62167-console-serving-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:13.765998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:13.765769    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4jzd\" (UniqueName: \"kubernetes.io/projected/3ec3fa67-9a08-46d3-8274-cdf144a62167-kube-api-access-t4jzd\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:54:14.357453 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.357416    5784 generic.go:356] "Generic (PLEG): container finished" podID="3ec3fa67-9a08-46d3-8274-cdf144a62167" containerID="2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2" exitCode=0
Nov 03 11:54:14.357627 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.357468    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79d4798df4-pzlw5"
Nov 03 11:54:14.357627 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.357486    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4798df4-pzlw5" event={"ID":"3ec3fa67-9a08-46d3-8274-cdf144a62167","Type":"ContainerDied","Data":"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"}
Nov 03 11:54:14.357627 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.357527    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79d4798df4-pzlw5" event={"ID":"3ec3fa67-9a08-46d3-8274-cdf144a62167","Type":"ContainerDied","Data":"a501f3ce1a525084ac91b6b1e4a44cc689d035b99d868799f6bc2ea2ca4d067a"}
Nov 03 11:54:14.357627 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.357547    5784 scope.go:122] "RemoveContainer" containerID="2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"
Nov 03 11:54:14.365529 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.365504    5784 scope.go:122] "RemoveContainer" containerID="2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"
Nov 03 11:54:14.365851 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:54:14.365827    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2\": container with ID starting with 2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2 not found: ID does not exist" containerID="2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"
Nov 03 11:54:14.365898 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.365867    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2"} err="failed to get container status \"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2\": rpc error: code = NotFound desc = could not find container \"2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2\": container with ID starting with 2849e7427d875b8ea9eb5ea09d2d92b0a0aa36f8a275162838da5333b7bbd1a2 not found: ID does not exist"
Nov 03 11:54:14.508409 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.508359    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:54:14.554243 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:14.554205    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79d4798df4-pzlw5"]
Nov 03 11:54:15.678971 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:15.678930    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec3fa67-9a08-46d3-8274-cdf144a62167" path="/var/lib/kubelet/pods/3ec3fa67-9a08-46d3-8274-cdf144a62167/volumes"
Nov 03 11:54:26.469797 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:54:26.469757    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:54:41.719409 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:54:41.719315    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:54:56.470866 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:54:56.470820    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:55:26.470276 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:55:26.470230    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:55:36.412034 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:36.412001    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:36.642525 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:36.642490    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:37.363086 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.363051    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"]
Nov 03 11:55:37.363520 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.363503    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3ec3fa67-9a08-46d3-8274-cdf144a62167" containerName="console"
Nov 03 11:55:37.363570 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.363522    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec3fa67-9a08-46d3-8274-cdf144a62167" containerName="console"
Nov 03 11:55:37.363632 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.363622    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec3fa67-9a08-46d3-8274-cdf144a62167" containerName="console"
Nov 03 11:55:37.367034 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.367013    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.440007 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.439979    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.440399 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.440042    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.440399 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.440162    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxvq\" (UniqueName: \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.464111 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.464090    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 11:55:37.464232 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.464122    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 11:55:37.464466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.464449    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-mklvr\""
Nov 03 11:55:37.465136 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.465119    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"]
Nov 03 11:55:37.540870 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.540841    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.541056 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.540901    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.541056 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.540940    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxvq\" (UniqueName: \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.541330 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.541310    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.541417 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.541343    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.597620 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.597595    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxvq\" (UniqueName: \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.682446 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.682418    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:55:37.861893 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:37.861854    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:38.023169 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.023085    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:38.046189 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.046158    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"]
Nov 03 11:55:38.049650 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:38.049623    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b173c7e_8f0d_48ca_8780_78080bbd6bc7.slice/crio-cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5 WatchSource:0}: Error finding container cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5: Status 404 returned error can't find the container with id cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5
Nov 03 11:55:38.539026 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.538997    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"]
Nov 03 11:55:38.543484 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.543457    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.591549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.591518    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-nmdsf\""
Nov 03 11:55:38.622045 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.622017    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"]
Nov 03 11:55:38.634067 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.634044    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"]
Nov 03 11:55:38.637427 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.637411    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.650192 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.650168    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.650323 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.650207    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.650438 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.650413    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cktj\" (UniqueName: \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.655355 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.655332    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerStarted","Data":"cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5"}
Nov 03 11:55:38.685490 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.685467    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-l4wkf\""
Nov 03 11:55:38.690867 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.690845    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"]
Nov 03 11:55:38.741299 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.741270    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"]
Nov 03 11:55:38.744759 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.744737    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.750878 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750783    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cktj\" (UniqueName: \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.750878 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750826    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.750878 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750874    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.751055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750905    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.751055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750937    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.751055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.750967    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzkw\" (UniqueName: \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.751317 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.751297    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.751368 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.751328    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.793682 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.793606    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-mkbt6\""
Nov 03 11:55:38.799817 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.799786    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"]
Nov 03 11:55:38.807117 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.807088    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cktj\" (UniqueName: \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.840005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.839971    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"]
Nov 03 11:55:38.845020 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.844990    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:38.848244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.848216    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:38.852255 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852231    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6tj\" (UniqueName: \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.852350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852281    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.852350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852341    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.852464 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852398    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.852499 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852455    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.852535 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852496    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzkw\" (UniqueName: \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.852786 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852766    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.852866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.852808    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.860190 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.860173    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:55:38.895854 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.895823    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2d28f\""
Nov 03 11:55:38.897230 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.897147    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"]
Nov 03 11:55:38.905395 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.905338    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzkw\" (UniqueName: \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.952467 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.952442    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:55:38.953730 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953685    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6tj\" (UniqueName: \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.953857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953753    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.953857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953793    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.953857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953849    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:38.954015 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953895    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjz2\" (UniqueName: \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:38.954015 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.953957    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:38.954109 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.954092    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:38.954150 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:38.954133    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:39.008603 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.008367    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6tj\" (UniqueName: \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:39.054581 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.054492    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.054581 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.054538    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjz2\" (UniqueName: \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.054811 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.054636    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.054928 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.054904    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.054999 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.054981    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.060426 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.060396    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:55:39.107972 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.107940    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjz2\" (UniqueName: \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.162186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.162149    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:55:39.233202 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.233173    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"]
Nov 03 11:55:39.234464 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:39.234416    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcedc28e_f9f0_441a_af0d_22a47422e830.slice/crio-7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31 WatchSource:0}: Error finding container 7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31: Status 404 returned error can't find the container with id 7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31
Nov 03 11:55:39.324836 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.324761    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"]
Nov 03 11:55:39.329042 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:39.329013    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd5e9db_c856_4a44_b255_cebe5c51d2c5.slice/crio-57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf WatchSource:0}: Error finding container 57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf: Status 404 returned error can't find the container with id 57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf
Nov 03 11:55:39.427055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.427026    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"]
Nov 03 11:55:39.430218 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:39.430191    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a52f0f_e1e4_42ec_b08f_e95b8aa897b2.slice/crio-7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a WatchSource:0}: Error finding container 7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a: Status 404 returned error can't find the container with id 7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a
Nov 03 11:55:39.529000 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.528865    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"]
Nov 03 11:55:39.532446 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:39.532415    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfd1ae8_6484_4158_a037_9ec66201e048.slice/crio-38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220 WatchSource:0}: Error finding container 38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220: Status 404 returned error can't find the container with id 38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220
Nov 03 11:55:39.661290 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.661252    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85" event={"ID":"dbfd1ae8-6484-4158-a037-9ec66201e048","Type":"ContainerStarted","Data":"38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220"}
Nov 03 11:55:39.662750 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.662718    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx" event={"ID":"bcedc28e-f9f0-441a-af0d-22a47422e830","Type":"ContainerStarted","Data":"7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31"}
Nov 03 11:55:39.664008 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.663984    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerStarted","Data":"7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a"}
Nov 03 11:55:39.665364 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.665342    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q" event={"ID":"efd5e9db-c856-4a44-b255-cebe5c51d2c5","Type":"ContainerStarted","Data":"57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf"}
Nov 03 11:55:39.816393 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.816347    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"]
Nov 03 11:55:39.821357 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.821335    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.861480 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.861441    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9nj9\" (UniqueName: \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.861630 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.861499    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.861681 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.861628    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.872392 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.872351    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-n7g4h\""
Nov 03 11:55:39.873305 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.873260    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"]
Nov 03 11:55:39.962746 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.962671    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.962746 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.962726    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9nj9\" (UniqueName: \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.963036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.962778    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.963155 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.963132    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:39.963259 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:39.963158    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:40.017644 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:40.017612    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9nj9\" (UniqueName: \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:40.141916 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:40.141878    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:55:40.527730 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:40.527647    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"]
Nov 03 11:55:40.532944 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:40.532903    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd502fc9_219b_42d1_81d1_c4a3f88738f0.slice/crio-d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed WatchSource:0}: Error finding container d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed: Status 404 returned error can't find the container with id d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed
Nov 03 11:55:40.675394 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:40.675340    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerStarted","Data":"d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed"}
Nov 03 11:55:41.149768 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.149734    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"]
Nov 03 11:55:41.153788 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.153760    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.248387 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.248344    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-47mdz\""
Nov 03 11:55:41.249785 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.249761    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"]
Nov 03 11:55:41.276920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.276884    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llt7l\" (UniqueName: \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.277094 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.276959    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.277094 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.277060    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.377994 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.377952    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llt7l\" (UniqueName: \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.378184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.378096    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.378184 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.378167    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.378718 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.378659    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.379309 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.379286    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.443863 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.439072    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llt7l\" (UniqueName: \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.475346 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.474990    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:55:41.892370 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:41.891948    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"]
Nov 03 11:55:41.903590 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:41.903552    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d21c781_3ef6_47f3_9106_cb6cf09fa6f9.slice/crio-1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f WatchSource:0}: Error finding container 1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f: Status 404 returned error can't find the container with id 1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f
Nov 03 11:55:42.053164 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:42.053130    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:42.693923 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:42.693840    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs" event={"ID":"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9","Type":"ContainerStarted","Data":"1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f"}
Nov 03 11:55:43.637871 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.635665    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"]
Nov 03 11:55:43.644433 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.643234    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.694887 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.694796    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"]
Nov 03 11:55:43.703359 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.703332    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.703630 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.703603    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zd9z\" (UniqueName: \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.703720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.703668    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.806712 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.805166    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.806712 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.805251    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zd9z\" (UniqueName: \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.806712 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.805296    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.806712 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.805836    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.806712 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.806085    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.863777 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.863711    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zd9z\" (UniqueName: \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:43.969435 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:43.969302    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:55:44.396586 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.396551    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"]
Nov 03 11:55:44.400305 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:44.400278    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef52971_6bf7_43b3_890e_f2752b770cb2.slice/crio-2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c WatchSource:0}: Error finding container 2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c: Status 404 returned error can't find the container with id 2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c
Nov 03 11:55:44.448397 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.445731    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:44.596734 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.596699    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:44.727003 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.726881    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerStarted","Data":"2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c"}
Nov 03 11:55:44.849962 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.849925    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 11:55:44.856950 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.856924    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919556    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919606    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fv2\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919637    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919661    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919692    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919721    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919774    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919826    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.920083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.919880    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:44.962426 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.961086    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 11:55:44.962426 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.961087    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 11:55:44.962426 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.961156    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 11:55:44.962426 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.961214    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 11:55:44.962843 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:44.962716    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 11:55:45.011123 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.011051    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:45.021114 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021087    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021148    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021180    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021200    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fv2\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021223    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021240    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021275 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021260    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021708 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021309    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021708 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021344    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021708 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021542    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.021922 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.021710    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.022314 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.022293    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.022449 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.022423    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.022601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.022577    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.030738 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.030711    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.030991 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.030906    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.082538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.082500    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fv2\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.086853 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.086444    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-6584f\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.157876 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.157835    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"]
Nov 03 11:55:45.164601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.163995    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.178313 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.178126    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:55:45.188793 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.188718    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:45.220877 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.220798    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"]
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224281    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224349    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224400    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224429    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627077e6-5541-4411-b611-a41dcd52ed1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224473    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224517    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfns9\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-kube-api-access-tfns9\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224575    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224612    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.224879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.224672    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627077e6-5541-4411-b611-a41dcd52ed1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.229837 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.229793    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-hfbh2\""
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330008    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330056    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330089    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627077e6-5541-4411-b611-a41dcd52ed1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330138    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330195    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfns9\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-kube-api-access-tfns9\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330240    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330275    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330321    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627077e6-5541-4411-b611-a41dcd52ed1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.330361    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.331677    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.332347    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627077e6-5541-4411-b611-a41dcd52ed1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.332632    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.332890    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.333550 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.333128    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.334789 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.334615    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627077e6-5541-4411-b611-a41dcd52ed1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.342549 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.342507    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627077e6-5541-4411-b611-a41dcd52ed1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.369494 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.368434    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:45.393094 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.392796    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfns9\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-kube-api-access-tfns9\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.397320 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.394572    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627077e6-5541-4411-b611-a41dcd52ed1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6\" (UID: \"627077e6-5541-4411-b611-a41dcd52ed1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.487622 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.487584    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:55:45.687398 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.687311    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 11:55:45.746607 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.746527    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerStarted","Data":"e0a4d2bef09aaedb77833e47c397bdef127a4ae5c02bbb038c262c1a9ac285a9"}
Nov 03 11:55:45.892401 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:45.892303    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"]
Nov 03 11:55:45.904089 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:45.904055    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627077e6_5541_4411_b611_a41dcd52ed1b.slice/crio-b3968531cbefbab81d46b6cc84a65ee070b118d720bff5ac46c8be55c94558d1 WatchSource:0}: Error finding container b3968531cbefbab81d46b6cc84a65ee070b118d720bff5ac46c8be55c94558d1: Status 404 returned error can't find the container with id b3968531cbefbab81d46b6cc84a65ee070b118d720bff5ac46c8be55c94558d1
Nov 03 11:55:46.015648 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.013282    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"]
Nov 03 11:55:46.020141 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.020116    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.039309 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.039283    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"]
Nov 03 11:55:46.043670 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.043649    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.068596 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.068574    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"]
Nov 03 11:55:46.075361 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.075334    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"]
Nov 03 11:55:46.075631 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.075565    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.091693 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.091674    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"]
Nov 03 11:55:46.096059 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.096028    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.126447 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.126418    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"]
Nov 03 11:55:46.141220 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141192    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.141406 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141334    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vrd9\" (UniqueName: \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.141406 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141372    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.141538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141430    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x677x\" (UniqueName: \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.141538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141464    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.141538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141492    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.141677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141563    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.141677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141607    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.141774 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.141716    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbs64\" (UniqueName: \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.143913 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.143780    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"]
Nov 03 11:55:46.146892 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.146826    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"]
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243034    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbs64\" (UniqueName: \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243087    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243121    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243150    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vrd9\" (UniqueName: \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243181    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243215    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x677x\" (UniqueName: \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243243    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243272    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243300    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhgw\" (UniqueName: \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243367    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243407    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243452    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.243966    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.244525    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.245101    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.245629    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.245863    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.246248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.246119    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.303648 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.303568    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x677x\" (UniqueName: \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.307889 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.307832    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbs64\" (UniqueName: \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.315029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.315005    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vrd9\" (UniqueName: \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.344857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.344123    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.344857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.344201    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhgw\" (UniqueName: \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.344857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.344259    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.344857 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.344726    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.345881 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.345776    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.367747 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.367483    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:55:46.376387 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.376345    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:55:46.397290 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.397258    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:55:46.404106 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.404081    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhgw\" (UniqueName: \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.417716 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.417687    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:55:46.664026 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.663867    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:46.759089 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.758998    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6" event={"ID":"627077e6-5541-4411-b611-a41dcd52ed1b","Type":"ContainerStarted","Data":"b3968531cbefbab81d46b6cc84a65ee070b118d720bff5ac46c8be55c94558d1"}
Nov 03 11:55:46.806660 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.806590    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"]
Nov 03 11:55:46.821094 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.821060    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:46.851061 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:46.851017    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb55914ce_96c7_493f_a326_ea331fc7af6f.slice/crio-be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb WatchSource:0}: Error finding container be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb: Status 404 returned error can't find the container with id be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb
Nov 03 11:55:46.859869 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.858474    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"]
Nov 03 11:55:46.908783 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.908597    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"]
Nov 03 11:55:46.908783 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:46.908755    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0861f99c_f3f6_4049_8b00_a761e99740f6.slice/crio-9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24 WatchSource:0}: Error finding container 9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24: Status 404 returned error can't find the container with id 9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24
Nov 03 11:55:46.910684 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:46.910513    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"]
Nov 03 11:55:46.917652 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:55:46.917626    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8293e55a_9e7f_4951_9c94_45919d6674e7.slice/crio-67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908 WatchSource:0}: Error finding container 67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908: Status 404 returned error can't find the container with id 67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908
Nov 03 11:55:47.782952 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:47.778717    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg" event={"ID":"0861f99c-f3f6-4049-8b00-a761e99740f6","Type":"ContainerStarted","Data":"9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24"}
Nov 03 11:55:47.782952 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:47.781739    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6" event={"ID":"b47fbddc-b469-43dd-aeae-7ef507e47d2f","Type":"ContainerStarted","Data":"9e61a437ae233febc7a0857e7236978eeb37a604e372f93265aa1d853a727f0b"}
Nov 03 11:55:47.788008 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:47.787970    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh" event={"ID":"8293e55a-9e7f-4951-9c94-45919d6674e7","Type":"ContainerStarted","Data":"67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908"}
Nov 03 11:55:47.791624 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:47.791600    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerStarted","Data":"be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb"}
Nov 03 11:55:49.161133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:49.161068    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:55:56.503423 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:55:56.503305    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:55:56.974409 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:56.974321    5784 generic.go:356] "Generic (PLEG): container finished" podID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerID="0ebbffbd9162fb29da8e7cd3b32c839ac1c8cb1c2bdbc15a87eba942e77f3272" exitCode=0
Nov 03 11:55:56.975767 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:56.974761    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerDied","Data":"0ebbffbd9162fb29da8e7cd3b32c839ac1c8cb1c2bdbc15a87eba942e77f3272"}
Nov 03 11:55:59.762636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:59.762470    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:55:59.762636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:59.762562    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:55:59.762636 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:55:59.762605    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:56:00.041866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.041729    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6" event={"ID":"627077e6-5541-4411-b611-a41dcd52ed1b","Type":"ContainerStarted","Data":"e292707b8cfedd7e263921cb29b1aaf877e3a01e3583b14a1db512ed2c907157"}
Nov 03 11:56:00.050243 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.050201    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerStarted","Data":"61dd4e8f8d74ef30c262fde4306528953f784a975b474941f5d2711c97f7244f"}
Nov 03 11:56:00.281840 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.281771    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6" podStartSLOduration=1.427497024 podStartE2EDuration="15.281750625s" totalImagesPullingTime="13.854253601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:00.171873442 +0000 UTC m=+266.614075762" watchObservedRunningTime="2026-11-03 11:56:00.281750625 +0000 UTC m=+266.723952931"
Nov 03 11:56:00.454512 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.454464    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:56:00.454647 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.454570    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:56:00.454647 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.454616    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:56:00.493413 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.488858    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:56:00.493413 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.489262    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:56:00.495763 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:00.495712    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:56:01.065129 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.064290    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerStarted","Data":"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"}
Nov 03 11:56:01.070521 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.070003    5784 generic.go:356] "Generic (PLEG): container finished" podID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerID="a56bab75a343409453b2a8ac62c1178c129665608f282b7a6e300d8efb6caa6b" exitCode=0
Nov 03 11:56:01.070521 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.070103    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx" event={"ID":"bcedc28e-f9f0-441a-af0d-22a47422e830","Type":"ContainerDied","Data":"a56bab75a343409453b2a8ac62c1178c129665608f282b7a6e300d8efb6caa6b"}
Nov 03 11:56:01.077673 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.077646    5784 generic.go:356] "Generic (PLEG): container finished" podID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerID="61dd4e8f8d74ef30c262fde4306528953f784a975b474941f5d2711c97f7244f" exitCode=0
Nov 03 11:56:01.077994 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.077898    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerDied","Data":"61dd4e8f8d74ef30c262fde4306528953f784a975b474941f5d2711c97f7244f"}
Nov 03 11:56:01.079502 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.079483    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6"
Nov 03 11:56:01.178761 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.178732    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:56:01.184749 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.184722    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:56:01.297967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:01.292342    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podStartSLOduration=2.53253774 podStartE2EDuration="17.292324s" totalImagesPullingTime="14.75978626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:01.175965748 +0000 UTC m=+267.618168056" watchObservedRunningTime="2026-11-03 11:56:01.292324 +0000 UTC m=+267.734526306"
Nov 03 11:56:02.087091 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.087063    5784 generic.go:356] "Generic (PLEG): container finished" podID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerID="cd242eb132907fb8e1e9a0e9c1d8a6849a2f7576cd5b4575ffd20d3225bdef6c" exitCode=0
Nov 03 11:56:02.087504 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.087444    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85" event={"ID":"dbfd1ae8-6484-4158-a037-9ec66201e048","Type":"ContainerDied","Data":"cd242eb132907fb8e1e9a0e9c1d8a6849a2f7576cd5b4575ffd20d3225bdef6c"}
Nov 03 11:56:02.089924 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.089902    5784 generic.go:356] "Generic (PLEG): container finished" podID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerID="28030237a2b183aca5e7c843f9025e69e094ddad5a9ee0fb4ffa4d6bce08ac6f" exitCode=0
Nov 03 11:56:02.090029 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.090003    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q" event={"ID":"efd5e9db-c856-4a44-b255-cebe5c51d2c5","Type":"ContainerDied","Data":"28030237a2b183aca5e7c843f9025e69e094ddad5a9ee0fb4ffa4d6bce08ac6f"}
Nov 03 11:56:02.093939 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.093913    5784 generic.go:356] "Generic (PLEG): container finished" podID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerID="a058b2091cd6dc5caa249079229d415293106dcc6c540cf9108e5efd0400f40a" exitCode=0
Nov 03 11:56:02.094853 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.094831    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg" event={"ID":"0861f99c-f3f6-4049-8b00-a761e99740f6","Type":"ContainerDied","Data":"a058b2091cd6dc5caa249079229d415293106dcc6c540cf9108e5efd0400f40a"}
Nov 03 11:56:02.094944 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.094875    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:56:02.096167 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:02.096095    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:56:03.103022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.102979    5784 generic.go:356] "Generic (PLEG): container finished" podID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerID="f3611c70f326cf6a7d73e0cfd7e9e1ea2415563026f3243f62f64286e8153403" exitCode=0
Nov 03 11:56:03.103525 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.103137    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerDied","Data":"f3611c70f326cf6a7d73e0cfd7e9e1ea2415563026f3243f62f64286e8153403"}
Nov 03 11:56:03.107088 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.106489    5784 generic.go:356] "Generic (PLEG): container finished" podID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerID="df647115aac51c2f4fabef60fa72163a486328b1a2e81bd96d585c76d3726a65" exitCode=0
Nov 03 11:56:03.107088 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.106601    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerDied","Data":"df647115aac51c2f4fabef60fa72163a486328b1a2e81bd96d585c76d3726a65"}
Nov 03 11:56:03.109531 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.109510    5784 generic.go:356] "Generic (PLEG): container finished" podID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerID="d1d4071274c7df54962cbecad3a7631a3f6c66f9ad55f0e4a1b671d25ea8dfa8" exitCode=0
Nov 03 11:56:03.109615 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.109586    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6" event={"ID":"b47fbddc-b469-43dd-aeae-7ef507e47d2f","Type":"ContainerDied","Data":"d1d4071274c7df54962cbecad3a7631a3f6c66f9ad55f0e4a1b671d25ea8dfa8"}
Nov 03 11:56:03.113601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.113251    5784 generic.go:356] "Generic (PLEG): container finished" podID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerID="93b6eaa7a90c6d77f7ba1b2bb2d80b2eb62fa17737c106644c383c44a83472e8" exitCode=0
Nov 03 11:56:03.113601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.113326    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh" event={"ID":"8293e55a-9e7f-4951-9c94-45919d6674e7","Type":"ContainerDied","Data":"93b6eaa7a90c6d77f7ba1b2bb2d80b2eb62fa17737c106644c383c44a83472e8"}
Nov 03 11:56:03.115893 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.115799    5784 generic.go:356] "Generic (PLEG): container finished" podID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerID="e051f8801bf910aa38f582c45bdf56ba2b2d837ec34d896c56ca21b57c1a9d71" exitCode=0
Nov 03 11:56:03.115893 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:03.115859    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerDied","Data":"e051f8801bf910aa38f582c45bdf56ba2b2d837ec34d896c56ca21b57c1a9d71"}
Nov 03 11:56:04.122429 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.122353    5784 generic.go:356] "Generic (PLEG): container finished" podID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerID="f0e6288050e98af8f9fe3d939de1c59874944a861911f0ac4ee4fa5738b16261" exitCode=0
Nov 03 11:56:04.122869 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.122466    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs" event={"ID":"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9","Type":"ContainerDied","Data":"f0e6288050e98af8f9fe3d939de1c59874944a861911f0ac4ee4fa5738b16261"}
Nov 03 11:56:04.125724 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.125703    5784 generic.go:356] "Generic (PLEG): container finished" podID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerID="9596187ca5b174174fd8b97a3514ba422152f860d81bc722e66f792edb723456" exitCode=0
Nov 03 11:56:04.125817 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.125787    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx" event={"ID":"bcedc28e-f9f0-441a-af0d-22a47422e830","Type":"ContainerDied","Data":"9596187ca5b174174fd8b97a3514ba422152f860d81bc722e66f792edb723456"}
Nov 03 11:56:04.128147 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.128129    5784 generic.go:356] "Generic (PLEG): container finished" podID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerID="23daa7afe935eea19f6b372909a5dd3c2ed09d425bb8972d30fdabe25888b40f" exitCode=0
Nov 03 11:56:04.128218 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.128187    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerDied","Data":"23daa7afe935eea19f6b372909a5dd3c2ed09d425bb8972d30fdabe25888b40f"}
Nov 03 11:56:04.130293 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.130275    5784 generic.go:356] "Generic (PLEG): container finished" podID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerID="d36e25823cde616d8b637761286f6a1ac6ab5d4dbcc86cfe23c12356836c9edf" exitCode=0
Nov 03 11:56:04.130388 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.130328    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerDied","Data":"d36e25823cde616d8b637761286f6a1ac6ab5d4dbcc86cfe23c12356836c9edf"}
Nov 03 11:56:04.132197 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.132178    5784 generic.go:356] "Generic (PLEG): container finished" podID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerID="e805b17bc85852c8e7a71f45b7be723359337835f574ffb2791e63c789ff8d91" exitCode=0
Nov 03 11:56:04.132289 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.132262    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q" event={"ID":"efd5e9db-c856-4a44-b255-cebe5c51d2c5","Type":"ContainerDied","Data":"e805b17bc85852c8e7a71f45b7be723359337835f574ffb2791e63c789ff8d91"}
Nov 03 11:56:04.134228 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.134193    5784 generic.go:356] "Generic (PLEG): container finished" podID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerID="ec9f40aac01785beee864b25e2700cfe770b6a2fff62f5e43b0f8ecd01cce514" exitCode=0
Nov 03 11:56:04.134296 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:04.134250    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg" event={"ID":"0861f99c-f3f6-4049-8b00-a761e99740f6","Type":"ContainerDied","Data":"ec9f40aac01785beee864b25e2700cfe770b6a2fff62f5e43b0f8ecd01cce514"}
Nov 03 11:56:05.149078 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.148021    5784 generic.go:356] "Generic (PLEG): container finished" podID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerID="28782e9d2ab6a405f5c40c7f90d1a05b1b99458ebd0822697215da78ecc54a31" exitCode=0
Nov 03 11:56:05.149078 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.148150    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerDied","Data":"28782e9d2ab6a405f5c40c7f90d1a05b1b99458ebd0822697215da78ecc54a31"}
Nov 03 11:56:05.157826 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.156997    5784 generic.go:356] "Generic (PLEG): container finished" podID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerID="c6c0f98d88d1a0cc28c5eebcd3de3d5a91b763d05ccc1ec4c546b7439e4d32d3" exitCode=0
Nov 03 11:56:05.157826 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.157094    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85" event={"ID":"dbfd1ae8-6484-4158-a037-9ec66201e048","Type":"ContainerDied","Data":"c6c0f98d88d1a0cc28c5eebcd3de3d5a91b763d05ccc1ec4c546b7439e4d32d3"}
Nov 03 11:56:05.167154 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.167124    5784 generic.go:356] "Generic (PLEG): container finished" podID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerID="d210109b291d4a4af4a13beb9bceda27843ead123e742c8ab15fc856694fe267" exitCode=0
Nov 03 11:56:05.167268 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.167233    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6" event={"ID":"b47fbddc-b469-43dd-aeae-7ef507e47d2f","Type":"ContainerDied","Data":"d210109b291d4a4af4a13beb9bceda27843ead123e742c8ab15fc856694fe267"}
Nov 03 11:56:05.174222 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.173627    5784 generic.go:356] "Generic (PLEG): container finished" podID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerID="2990675d5f4648ca997fb95ae3004c202dd135d7de549dbe329ef90bd1a953a6" exitCode=0
Nov 03 11:56:05.174222 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:05.173679    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerDied","Data":"2990675d5f4648ca997fb95ae3004c202dd135d7de549dbe329ef90bd1a953a6"}
Nov 03 11:56:06.196341 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:06.196299    5784 generic.go:356] "Generic (PLEG): container finished" podID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerID="3e33c295a510f8d0661095ce6a5fc10b3da796fdf92f0cf3e8acc4af8006fdfe" exitCode=0
Nov 03 11:56:06.196879 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:06.196445    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs" event={"ID":"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9","Type":"ContainerDied","Data":"3e33c295a510f8d0661095ce6a5fc10b3da796fdf92f0cf3e8acc4af8006fdfe"}
Nov 03 11:56:06.211873 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:06.211842    5784 generic.go:356] "Generic (PLEG): container finished" podID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerID="55aacd96130e78507109f83f53ff2a08db787fdfbf0bbe9a5375508c22c56f58" exitCode=0
Nov 03 11:56:06.212036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:06.211895    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh" event={"ID":"8293e55a-9e7f-4951-9c94-45919d6674e7","Type":"ContainerDied","Data":"55aacd96130e78507109f83f53ff2a08db787fdfbf0bbe9a5375508c22c56f58"}
Nov 03 11:56:11.319501 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:11.319335    5784 generic.go:356] "Generic (PLEG): container finished" podID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerID="57fa9e1e2c2cfe3a21d7e4132950539362bab75b79ccfa8b1d715782051e97b3" exitCode=0
Nov 03 11:56:11.320516 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:11.320480    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerDied","Data":"57fa9e1e2c2cfe3a21d7e4132950539362bab75b79ccfa8b1d715782051e97b3"}
Nov 03 11:56:12.970635 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:12.969955    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:26.518599 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:56:26.518553    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:56:28.598780 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:28.597838    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerStarted","Data":"d5cad450fd5725360bbea4002761b4dcdc4fcc2d7344459dc7a960248380a033"}
Nov 03 11:56:28.603341 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:28.602772    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerStarted","Data":"7a44b863a2364142edb59cb182aceda6a0886a4fb122677190f0a32a3b4e15eb"}
Nov 03 11:56:28.707139 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:28.707092    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" podStartSLOduration=2.821141513 podStartE2EDuration="51.707080091s" totalImagesPullingTime="48.885938578s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:28.706298865 +0000 UTC m=+295.148501187" watchObservedRunningTime="2026-11-03 11:56:28.707080091 +0000 UTC m=+295.149282393"
Nov 03 11:56:28.814407 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:28.813958    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" podStartSLOduration=3.165954647 podStartE2EDuration="45.813936416s" totalImagesPullingTime="42.647981769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:28.812228943 +0000 UTC m=+295.254431252" watchObservedRunningTime="2026-11-03 11:56:28.813936416 +0000 UTC m=+295.256138723"
Nov 03 11:56:29.614277 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.614240    5784 generic.go:356] "Generic (PLEG): container finished" podID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerID="b3ca994a7a3d0c3d478557f4f09f31cbb1249e7b19d94c3ef1f4317da66c34eb" exitCode=0
Nov 03 11:56:29.614744 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.614415    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg" event={"ID":"0861f99c-f3f6-4049-8b00-a761e99740f6","Type":"ContainerDied","Data":"b3ca994a7a3d0c3d478557f4f09f31cbb1249e7b19d94c3ef1f4317da66c34eb"}
Nov 03 11:56:29.618001 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.617975    5784 generic.go:356] "Generic (PLEG): container finished" podID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerID="4d7f8e045e5e00d1f16a1f6c4a0fd87f24e59d4ffa966c79e3ac0b2283a5e40a" exitCode=0
Nov 03 11:56:29.618105 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.618054    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx" event={"ID":"bcedc28e-f9f0-441a-af0d-22a47422e830","Type":"ContainerDied","Data":"4d7f8e045e5e00d1f16a1f6c4a0fd87f24e59d4ffa966c79e3ac0b2283a5e40a"}
Nov 03 11:56:29.620792 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.620768    5784 generic.go:356] "Generic (PLEG): container finished" podID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerID="d5cad450fd5725360bbea4002761b4dcdc4fcc2d7344459dc7a960248380a033" exitCode=0
Nov 03 11:56:29.620885 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.620858    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerDied","Data":"d5cad450fd5725360bbea4002761b4dcdc4fcc2d7344459dc7a960248380a033"}
Nov 03 11:56:29.623088 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.623065    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 11:56:29.624059 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.624036    5784 generic.go:356] "Generic (PLEG): container finished" podID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerID="7a44b863a2364142edb59cb182aceda6a0886a4fb122677190f0a32a3b4e15eb" exitCode=1
Nov 03 11:56:29.624149 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:29.624090    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerDied","Data":"7a44b863a2364142edb59cb182aceda6a0886a4fb122677190f0a32a3b4e15eb"}
Nov 03 11:56:30.632347 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.632306    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerStarted","Data":"90b7879a3399186f8a2207b54ea4a5e7fc9fd82aa3edf85870acd26775042610"}
Nov 03 11:56:30.636015 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.635988    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerStarted","Data":"e06e8c8f8ff214ba9e6dd36ae1e0d4dbaf65ef0b049cb3e6ce401f4be081a9e3"}
Nov 03 11:56:30.639494 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.639466    5784 generic.go:356] "Generic (PLEG): container finished" podID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerID="e009947d2269f546c36744b96e61e67b2522ba4654b2b9e0cb49980a58e11b6f" exitCode=0
Nov 03 11:56:30.639610 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.639575    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q" event={"ID":"efd5e9db-c856-4a44-b255-cebe5c51d2c5","Type":"ContainerDied","Data":"e009947d2269f546c36744b96e61e67b2522ba4654b2b9e0cb49980a58e11b6f"}
Nov 03 11:56:30.643842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.643818    5784 generic.go:356] "Generic (PLEG): container finished" podID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerID="aea1b70c94984768dc9dabd9f55edee8cc752e3f0b000c072871085bda01bc92" exitCode=0
Nov 03 11:56:30.643962 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.643937    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85" event={"ID":"dbfd1ae8-6484-4158-a037-9ec66201e048","Type":"ContainerDied","Data":"aea1b70c94984768dc9dabd9f55edee8cc752e3f0b000c072871085bda01bc92"}
Nov 03 11:56:30.751420 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.750321    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" podStartSLOduration=3.852771435 podStartE2EDuration="45.750300926s" totalImagesPullingTime="41.897529491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:30.74816467 +0000 UTC m=+297.190367003" watchObservedRunningTime="2026-11-03 11:56:30.750300926 +0000 UTC m=+297.192503228"
Nov 03 11:56:30.860176 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.860147    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 11:56:30.879009 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.878984    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:56:30.885896 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.885875    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:56:30.942641 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.939955    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:56:30.949171 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.949140    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:56:30.963660 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.963609    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" podStartSLOduration=3.233911166 podStartE2EDuration="52.963590555s" totalImagesPullingTime="49.729679389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:30.9623941 +0000 UTC m=+297.404596422" watchObservedRunningTime="2026-11-03 11:56:30.963590555 +0000 UTC m=+297.405792863"
Nov 03 11:56:30.973524 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973497    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\" (UniqueName: \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\") pod \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") "
Nov 03 11:56:30.973830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973551    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\") pod \"7ef52971-6bf7-43b3-890e-f2752b770cb2\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") "
Nov 03 11:56:30.973830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973597    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\" (UniqueName: \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\") pod \"7ef52971-6bf7-43b3-890e-f2752b770cb2\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") "
Nov 03 11:56:30.973830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973648    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\") pod \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") "
Nov 03 11:56:30.973830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973718    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\") pod \"7ef52971-6bf7-43b3-890e-f2752b770cb2\" (UID: \"7ef52971-6bf7-43b3-890e-f2752b770cb2\") "
Nov 03 11:56:30.973830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.973749    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\") pod \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\" (UID: \"2b173c7e-8f0d-48ca-8780-78080bbd6bc7\") "
Nov 03 11:56:30.976530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.976495    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle" pod "7ef52971-6bf7-43b3-890e-f2752b770cb2" (UID: "7ef52971-6bf7-43b3-890e-f2752b770cb2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:30.978326 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.978300    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq" pod "2b173c7e-8f0d-48ca-8780-78080bbd6bc7" (UID: "2b173c7e-8f0d-48ca-8780-78080bbd6bc7"). InnerVolumeSpecName "kube-api-access-lhxvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:30.979047 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.979019    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle" pod "2b173c7e-8f0d-48ca-8780-78080bbd6bc7" (UID: "2b173c7e-8f0d-48ca-8780-78080bbd6bc7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:30.981115 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.981068    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util" pod "2b173c7e-8f0d-48ca-8780-78080bbd6bc7" (UID: "2b173c7e-8f0d-48ca-8780-78080bbd6bc7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:30.982515 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.982421    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z" pod "7ef52971-6bf7-43b3-890e-f2752b770cb2" (UID: "7ef52971-6bf7-43b3-890e-f2752b770cb2"). InnerVolumeSpecName "kube-api-access-2zd9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:30.985561 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:30.985532    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util" pod "7ef52971-6bf7-43b3-890e-f2752b770cb2" (UID: "7ef52971-6bf7-43b3-890e-f2752b770cb2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074345    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\" (UniqueName: \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\") pod \"bcedc28e-f9f0-441a-af0d-22a47422e830\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074430    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\" (UniqueName: \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\") pod \"0861f99c-f3f6-4049-8b00-a761e99740f6\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074517    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\") pod \"bcedc28e-f9f0-441a-af0d-22a47422e830\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074550    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\") pod \"bcedc28e-f9f0-441a-af0d-22a47422e830\" (UID: \"bcedc28e-f9f0-441a-af0d-22a47422e830\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074573    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\") pod \"0861f99c-f3f6-4049-8b00-a761e99740f6\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074606    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\") pod \"0861f99c-f3f6-4049-8b00-a761e99740f6\" (UID: \"0861f99c-f3f6-4049-8b00-a761e99740f6\") "
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074805    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074821    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074836    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxvq\" (UniqueName: \"kubernetes.io/projected/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-kube-api-access-lhxvq\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074850    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ef52971-6bf7-43b3-890e-f2752b770cb2-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074864    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zd9z\" (UniqueName: \"kubernetes.io/projected/7ef52971-6bf7-43b3-890e-f2752b770cb2-kube-api-access-2zd9z\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.074876    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2b173c7e-8f0d-48ca-8780-78080bbd6bc7-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.075686 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.075647    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle" pod "bcedc28e-f9f0-441a-af0d-22a47422e830" (UID: "bcedc28e-f9f0-441a-af0d-22a47422e830"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.076126 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.076094    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle" pod "0861f99c-f3f6-4049-8b00-a761e99740f6" (UID: "0861f99c-f3f6-4049-8b00-a761e99740f6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.078031 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.077985    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw" pod "0861f99c-f3f6-4049-8b00-a761e99740f6" (UID: "0861f99c-f3f6-4049-8b00-a761e99740f6"). InnerVolumeSpecName "kube-api-access-ddhgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:31.078127 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.078088    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj" pod "bcedc28e-f9f0-441a-af0d-22a47422e830" (UID: "bcedc28e-f9f0-441a-af0d-22a47422e830"). InnerVolumeSpecName "kube-api-access-5cktj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:31.081913 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.081888    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util" pod "bcedc28e-f9f0-441a-af0d-22a47422e830" (UID: "bcedc28e-f9f0-441a-af0d-22a47422e830"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.083621 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.083600    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util" pod "0861f99c-f3f6-4049-8b00-a761e99740f6" (UID: "0861f99c-f3f6-4049-8b00-a761e99740f6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176360    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176413    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bcedc28e-f9f0-441a-af0d-22a47422e830-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176423    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176435    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0861f99c-f3f6-4049-8b00-a761e99740f6-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176449    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cktj\" (UniqueName: \"kubernetes.io/projected/bcedc28e-f9f0-441a-af0d-22a47422e830-kube-api-access-5cktj\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.176489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.176463    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddhgw\" (UniqueName: \"kubernetes.io/projected/0861f99c-f3f6-4049-8b00-a761e99740f6-kube-api-access-ddhgw\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:31.408328 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.408251    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:31.567635 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.567602    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:31.651834 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.651691    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx" event={"ID":"2b173c7e-8f0d-48ca-8780-78080bbd6bc7","Type":"ContainerDied","Data":"cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5"}
Nov 03 11:56:31.651834 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.651778    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb57cd43561bde208119555c28425f3e3b970e069e450d131ffd79354732c2c5"
Nov 03 11:56:31.651834 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.651710    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651czbcwx"
Nov 03 11:56:31.653347 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.653328    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 11:56:31.654151 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.654131    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz"
Nov 03 11:56:31.654260 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.654147    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz" event={"ID":"7ef52971-6bf7-43b3-890e-f2752b770cb2","Type":"ContainerDied","Data":"2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c"}
Nov 03 11:56:31.654260 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.654193    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc18b1519c0adbcd19dfa27bd0b7f33b0f74f9c0f96368fee26e11de1c5413c"
Nov 03 11:56:31.656719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.656695    5784 generic.go:356] "Generic (PLEG): container finished" podID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerID="e06e8c8f8ff214ba9e6dd36ae1e0d4dbaf65ef0b049cb3e6ce401f4be081a9e3" exitCode=0
Nov 03 11:56:31.656829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.656773    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerDied","Data":"e06e8c8f8ff214ba9e6dd36ae1e0d4dbaf65ef0b049cb3e6ce401f4be081a9e3"}
Nov 03 11:56:31.662489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.662476    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg"
Nov 03 11:56:31.662583 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.662503    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6stfpg" event={"ID":"0861f99c-f3f6-4049-8b00-a761e99740f6","Type":"ContainerDied","Data":"9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24"}
Nov 03 11:56:31.662583 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.662545    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9691556c06f2fa94a438b58ad0b0fcbcf7fecdb4f7323a395ba82e0895ae3c24"
Nov 03 11:56:31.664838 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.664815    5784 generic.go:356] "Generic (PLEG): container finished" podID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerID="341d4fa58687522954f9f2b25663905f9da927565f5ed430f612ce31d18e3ae2" exitCode=0
Nov 03 11:56:31.664934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.664886    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs" event={"ID":"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9","Type":"ContainerDied","Data":"341d4fa58687522954f9f2b25663905f9da927565f5ed430f612ce31d18e3ae2"}
Nov 03 11:56:31.667195 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.667175    5784 generic.go:356] "Generic (PLEG): container finished" podID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerID="fe1bb796fe41e09e93f8380fec7b7d1dc364b02875e0c3cf9d4848ce1cc4714e" exitCode=0
Nov 03 11:56:31.667287 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.667262    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6" event={"ID":"b47fbddc-b469-43dd-aeae-7ef507e47d2f","Type":"ContainerDied","Data":"fe1bb796fe41e09e93f8380fec7b7d1dc364b02875e0c3cf9d4848ce1cc4714e"}
Nov 03 11:56:31.669666 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.669646    5784 generic.go:356] "Generic (PLEG): container finished" podID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerID="30c2bd69b1e340230264a6c0c48f2e930b14f4d6c4598f9882a267fff1db3680" exitCode=0
Nov 03 11:56:31.669763 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.669683    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh" event={"ID":"8293e55a-9e7f-4951-9c94-45919d6674e7","Type":"ContainerDied","Data":"30c2bd69b1e340230264a6c0c48f2e930b14f4d6c4598f9882a267fff1db3680"}
Nov 03 11:56:31.672040 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.672016    5784 generic.go:356] "Generic (PLEG): container finished" podID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerID="90b7879a3399186f8a2207b54ea4a5e7fc9fd82aa3edf85870acd26775042610" exitCode=0
Nov 03 11:56:31.672125 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.672089    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerDied","Data":"90b7879a3399186f8a2207b54ea4a5e7fc9fd82aa3edf85870acd26775042610"}
Nov 03 11:56:31.674737 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.674714    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx"
Nov 03 11:56:31.684478 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.684068    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c82tpx" event={"ID":"bcedc28e-f9f0-441a-af0d-22a47422e830","Type":"ContainerDied","Data":"7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31"}
Nov 03 11:56:31.684478 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.684109    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ea7150a9f12c2c797e381e84f3e45698586b22a0e25fa5e526ffd632472da31"
Nov 03 11:56:31.824727 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.824699    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:56:31.853246 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.853220    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:56:31.983616 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983585    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\") pod \"dbfd1ae8-6484-4158-a037-9ec66201e048\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") "
Nov 03 11:56:31.983805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983635    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\") pod \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") "
Nov 03 11:56:31.983805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983654    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\") pod \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") "
Nov 03 11:56:31.983805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983727    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\" (UniqueName: \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\") pod \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\" (UID: \"efd5e9db-c856-4a44-b255-cebe5c51d2c5\") "
Nov 03 11:56:31.983805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983780    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\") pod \"dbfd1ae8-6484-4158-a037-9ec66201e048\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") "
Nov 03 11:56:31.984012 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.983843    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\" (UniqueName: \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\") pod \"dbfd1ae8-6484-4158-a037-9ec66201e048\" (UID: \"dbfd1ae8-6484-4158-a037-9ec66201e048\") "
Nov 03 11:56:31.985411 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.984202    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle" pod "dbfd1ae8-6484-4158-a037-9ec66201e048" (UID: "dbfd1ae8-6484-4158-a037-9ec66201e048"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.986004 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.985982    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle" pod "efd5e9db-c856-4a44-b255-cebe5c51d2c5" (UID: "efd5e9db-c856-4a44-b255-cebe5c51d2c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.986236 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.986207    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2" pod "dbfd1ae8-6484-4158-a037-9ec66201e048" (UID: "dbfd1ae8-6484-4158-a037-9ec66201e048"). InnerVolumeSpecName "kube-api-access-bhjz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:31.986345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.986326    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw" pod "efd5e9db-c856-4a44-b255-cebe5c51d2c5" (UID: "efd5e9db-c856-4a44-b255-cebe5c51d2c5"). InnerVolumeSpecName "kube-api-access-hvzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:31.991673 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.991651    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util" pod "dbfd1ae8-6484-4158-a037-9ec66201e048" (UID: "dbfd1ae8-6484-4158-a037-9ec66201e048"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:31.992201 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:31.992178    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util" pod "efd5e9db-c856-4a44-b255-cebe5c51d2c5" (UID: "efd5e9db-c856-4a44-b255-cebe5c51d2c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.085039 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.084997    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjz2\" (UniqueName: \"kubernetes.io/projected/dbfd1ae8-6484-4158-a037-9ec66201e048-kube-api-access-bhjz2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.085039 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.085036    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.085244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.085052    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.085244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.085065    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/efd5e9db-c856-4a44-b255-cebe5c51d2c5-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.085244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.085078    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzkw\" (UniqueName: \"kubernetes.io/projected/efd5e9db-c856-4a44-b255-cebe5c51d2c5-kube-api-access-hvzkw\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.085244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.085091    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dbfd1ae8-6484-4158-a037-9ec66201e048-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.682732 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.682704    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q"
Nov 03 11:56:32.683134 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.682702    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6qgh4q" event={"ID":"efd5e9db-c856-4a44-b255-cebe5c51d2c5","Type":"ContainerDied","Data":"57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf"}
Nov 03 11:56:32.683134 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.682818    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57a120e53c9f49c3b80bc9dce54315db96383922a712f56169001b9ea3caadcf"
Nov 03 11:56:32.684737 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.684708    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85" event={"ID":"dbfd1ae8-6484-4158-a037-9ec66201e048","Type":"ContainerDied","Data":"38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220"}
Nov 03 11:56:32.684885 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.684744    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wnh85"
Nov 03 11:56:32.684885 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.684749    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38c73a614bc5b8b87838f82193819690e4251e4445d6b08e1ea79964a0ba8220"
Nov 03 11:56:32.687131 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.686935    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerStarted","Data":"4ee073cb35aea21635d45c16b6db833aa9c1f82bce16264f8945e8077bd39933"}
Nov 03 11:56:32.796865 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.796810    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" podStartSLOduration=3.783720249 podStartE2EDuration="53.796789717s" totalImagesPullingTime="50.013069468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:55:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:32.794504567 +0000 UTC m=+299.236706897" watchObservedRunningTime="2026-11-03 11:56:32.796789717 +0000 UTC m=+299.238992023"
Nov 03 11:56:32.844600 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.844564    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:56:32.887897 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.887868    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:56:32.892350 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.892331    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:56:32.898705 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.898683    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:56:32.901937 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.901919    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:56:32.993656 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993572    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\") pod \"8293e55a-9e7f-4951-9c94-45919d6674e7\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") "
Nov 03 11:56:32.993656 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993619    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\" (UniqueName: \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\") pod \"b55914ce-96c7-493f-a326-ea331fc7af6f\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") "
Nov 03 11:56:32.993656 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993655    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\") pod \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993683    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\" (UniqueName: \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\") pod \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993711    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\") pod \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993744    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\") pod \"b55914ce-96c7-493f-a326-ea331fc7af6f\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993772    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\" (UniqueName: \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\") pod \"8293e55a-9e7f-4951-9c94-45919d6674e7\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993809    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\") pod \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993847    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\") pod \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") "
Nov 03 11:56:32.993940 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993876    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\") pod \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") "
Nov 03 11:56:32.994286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.993955    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\") pod \"b55914ce-96c7-493f-a326-ea331fc7af6f\" (UID: \"b55914ce-96c7-493f-a326-ea331fc7af6f\") "
Nov 03 11:56:32.994286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994015    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\" (UniqueName: \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\") pod \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\" (UID: \"b47fbddc-b469-43dd-aeae-7ef507e47d2f\") "
Nov 03 11:56:32.994286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994043    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\") pod \"8293e55a-9e7f-4951-9c94-45919d6674e7\" (UID: \"8293e55a-9e7f-4951-9c94-45919d6674e7\") "
Nov 03 11:56:32.994286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994083    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\") pod \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\" (UID: \"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9\") "
Nov 03 11:56:32.994286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994143    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\" (UniqueName: \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\") pod \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\" (UID: \"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2\") "
Nov 03 11:56:32.994564 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994342    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle" pod "b47fbddc-b469-43dd-aeae-7ef507e47d2f" (UID: "b47fbddc-b469-43dd-aeae-7ef507e47d2f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.994564 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994469    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:32.994564 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.994485    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle" pod "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" (UID: "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.995730 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.995365    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle" pod "b55914ce-96c7-493f-a326-ea331fc7af6f" (UID: "b55914ce-96c7-493f-a326-ea331fc7af6f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.996805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.996687    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle" pod "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" (UID: "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.996805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.996714    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle" pod "8293e55a-9e7f-4951-9c94-45919d6674e7" (UID: "8293e55a-9e7f-4951-9c94-45919d6674e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:32.996805 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.996719    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64" pod "b55914ce-96c7-493f-a326-ea331fc7af6f" (UID: "b55914ce-96c7-493f-a326-ea331fc7af6f"). InnerVolumeSpecName "kube-api-access-gbs64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:32.997586 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.997272    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9" pod "8293e55a-9e7f-4951-9c94-45919d6674e7" (UID: "8293e55a-9e7f-4951-9c94-45919d6674e7"). InnerVolumeSpecName "kube-api-access-4vrd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:32.997586 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.997287    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l" pod "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" (UID: "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9"). InnerVolumeSpecName "kube-api-access-llt7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:32.997586 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.997326    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj" pod "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" (UID: "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2"). InnerVolumeSpecName "kube-api-access-2h6tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:32.998444 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:32.998420    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x" pod "b47fbddc-b469-43dd-aeae-7ef507e47d2f" (UID: "b47fbddc-b469-43dd-aeae-7ef507e47d2f"). InnerVolumeSpecName "kube-api-access-x677x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:33.002554 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.002531    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util" pod "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" (UID: "f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:33.004432 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.004408    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util" pod "b47fbddc-b469-43dd-aeae-7ef507e47d2f" (UID: "b47fbddc-b469-43dd-aeae-7ef507e47d2f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:33.004655 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.004628    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util" pod "b55914ce-96c7-493f-a326-ea331fc7af6f" (UID: "b55914ce-96c7-493f-a326-ea331fc7af6f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:33.005190 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.005159    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util" pod "8293e55a-9e7f-4951-9c94-45919d6674e7" (UID: "8293e55a-9e7f-4951-9c94-45919d6674e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:33.006096 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.006071    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util" pod "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" (UID: "2d21c781-3ef6-47f3-9106-cb6cf09fa6f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:33.095139 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095105    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h6tj\" (UniqueName: \"kubernetes.io/projected/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-kube-api-access-2h6tj\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095139 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095135    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095139 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095146    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbs64\" (UniqueName: \"kubernetes.io/projected/b55914ce-96c7-493f-a326-ea331fc7af6f-kube-api-access-gbs64\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095155    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095165    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llt7l\" (UniqueName: \"kubernetes.io/projected/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-kube-api-access-llt7l\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095173    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095181    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095191    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vrd9\" (UniqueName: \"kubernetes.io/projected/8293e55a-9e7f-4951-9c94-45919d6674e7-kube-api-access-4vrd9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095201    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b47fbddc-b469-43dd-aeae-7ef507e47d2f-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095209    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095217    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b55914ce-96c7-493f-a326-ea331fc7af6f-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095225    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x677x\" (UniqueName: \"kubernetes.io/projected/b47fbddc-b469-43dd-aeae-7ef507e47d2f-kube-api-access-x677x\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095235    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8293e55a-9e7f-4951-9c94-45919d6674e7-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.095391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.095242    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2d21c781-3ef6-47f3-9106-cb6cf09fa6f9-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:33.702308 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.702271    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh" event={"ID":"8293e55a-9e7f-4951-9c94-45919d6674e7","Type":"ContainerDied","Data":"67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908"}
Nov 03 11:56:33.702740 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.702367    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67de0ae0afd78a68532f403d0c7d1f1a87e1bc042a1f0a766f94be39e93bf908"
Nov 03 11:56:33.702740 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.702529    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4whnh"
Nov 03 11:56:33.705436 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.705408    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm" event={"ID":"b55914ce-96c7-493f-a326-ea331fc7af6f","Type":"ContainerDied","Data":"be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb"}
Nov 03 11:56:33.705551 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.705460    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be044da86acee7e5254585133d14827d9d3ddedcf95afd2d4400a7438cbb63cb"
Nov 03 11:56:33.705551 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.705463    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cg4xqm"
Nov 03 11:56:33.709475 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.709457    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh"
Nov 03 11:56:33.709800 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.709780    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apqwgh" event={"ID":"f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2","Type":"ContainerDied","Data":"7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a"}
Nov 03 11:56:33.709877 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.709818    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ff748b931a984464ad67a87dc740777127849533dc56ce1ecd87e7ec190ab1a"
Nov 03 11:56:33.712528 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.712505    5784 generic.go:356] "Generic (PLEG): container finished" podID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerID="4ee073cb35aea21635d45c16b6db833aa9c1f82bce16264f8945e8077bd39933" exitCode=0
Nov 03 11:56:33.712620 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.712570    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerDied","Data":"4ee073cb35aea21635d45c16b6db833aa9c1f82bce16264f8945e8077bd39933"}
Nov 03 11:56:33.714372 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.714351    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs" event={"ID":"2d21c781-3ef6-47f3-9106-cb6cf09fa6f9","Type":"ContainerDied","Data":"1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f"}
Nov 03 11:56:33.714372 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.714367    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71tx9gs"
Nov 03 11:56:33.714528 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.714400    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca4c332a2df717395d2862bf65843c95c655b966a52992e5aec8cef0f48573f"
Nov 03 11:56:33.716178 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.716155    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6" event={"ID":"b47fbddc-b469-43dd-aeae-7ef507e47d2f","Type":"ContainerDied","Data":"9e61a437ae233febc7a0857e7236978eeb37a604e372f93265aa1d853a727f0b"}
Nov 03 11:56:33.716286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.716193    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e61a437ae233febc7a0857e7236978eeb37a604e372f93265aa1d853a727f0b"
Nov 03 11:56:33.716286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.716264    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d79l4t6"
Nov 03 11:56:33.718010 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.717908    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 11:56:33.718010 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:33.717921    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 11:56:34.846273 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:34.846250    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:56:35.012147 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.012066    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\" (UniqueName: \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\") pod \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") "
Nov 03 11:56:35.012147 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.012117    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\") pod \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") "
Nov 03 11:56:35.012365 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.012210    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\") pod \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\" (UID: \"dd502fc9-219b-42d1-81d1-c4a3f88738f0\") "
Nov 03 11:56:35.012533 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.012512    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle" pod "dd502fc9-219b-42d1-81d1-c4a3f88738f0" (UID: "dd502fc9-219b-42d1-81d1-c4a3f88738f0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:35.014104 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.014084    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9" pod "dd502fc9-219b-42d1-81d1-c4a3f88738f0" (UID: "dd502fc9-219b-42d1-81d1-c4a3f88738f0"). InnerVolumeSpecName "kube-api-access-v9nj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:35.020525 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.020502    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util" pod "dd502fc9-219b-42d1-81d1-c4a3f88738f0" (UID: "dd502fc9-219b-42d1-81d1-c4a3f88738f0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:35.113388 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.113353    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:35.113547 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.113399    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9nj9\" (UniqueName: \"kubernetes.io/projected/dd502fc9-219b-42d1-81d1-c4a3f88738f0-kube-api-access-v9nj9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:35.113547 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.113411    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd502fc9-219b-42d1-81d1-c4a3f88738f0-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:35.726842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.726809    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446" event={"ID":"dd502fc9-219b-42d1-81d1-c4a3f88738f0","Type":"ContainerDied","Data":"d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed"}
Nov 03 11:56:35.726842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.726839    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64495446"
Nov 03 11:56:35.727030 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:35.726861    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d04694765def13bb0cc27ca78da51a32196f89e50d4ccf7507d8d9b675ffed"
Nov 03 11:56:36.212901 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:36.212868    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:36.367661 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:36.367627    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:37.942032 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942004    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"]
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942319    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942329    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942338    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942344    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942352    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942357    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942362    5784 cpu_manager.go:414] "containerMap: removing container" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942366    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942388    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942393    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942401    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942406    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942413    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="pull"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942417    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="pull"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942422    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942426    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="extract"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942432    5784 cpu_manager.go:414] "containerMap: removing container" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="util"
Nov 03 11:56:37.942430 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942437    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942442    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942446    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942453    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942457    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942462    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942467    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942478    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942482    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942487    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942491    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942497    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942501    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942507    5784 cpu_manager.go:414] "containerMap: removing container" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942511    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942518    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942523    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942529    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942533    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942539    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942543    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942552    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942556    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942561    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942564    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942572    5784 cpu_manager.go:414] "containerMap: removing container" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942576    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942581    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942585    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942595    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942599    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942605    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942610    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942615    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942619    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942626    5784 cpu_manager.go:414] "containerMap: removing container" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942630    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942636    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942639    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942644    5784 cpu_manager.go:414] "containerMap: removing container" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942649    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942657    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942662    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942667    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942671    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942677    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942681    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942692    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942697    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942702    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942706    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942711    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942715    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="util"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942720    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942724    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="pull"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942780    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b55914ce-96c7-493f-a326-ea331fc7af6f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942787    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0861f99c-f3f6-4049-8b00-a761e99740f6" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942792    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ef52971-6bf7-43b3-890e-f2752b770cb2" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942797    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47fbddc-b469-43dd-aeae-7ef507e47d2f" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942804    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b173c7e-8f0d-48ca-8780-78080bbd6bc7" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942810    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d21c781-3ef6-47f3-9106-cb6cf09fa6f9" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942815    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfd1ae8-6484-4158-a037-9ec66201e048" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942820    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a52f0f-e1e4-42ec-b08f-e95b8aa897b2" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942825    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd502fc9-219b-42d1-81d1-c4a3f88738f0" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942831    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8293e55a-9e7f-4951-9c94-45919d6674e7" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942837    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcedc28e-f9f0-441a-af0d-22a47422e830" containerName="extract"
Nov 03 11:56:37.942934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.942843    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd5e9db-c856-4a44-b255-cebe5c51d2c5" containerName="extract"
Nov 03 11:56:37.961533 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.961511    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:37.999848 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:37.999818    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"]
Nov 03 11:56:38.042764 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.042711    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b642\" (UniqueName: \"kubernetes.io/projected/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-kube-api-access-4b642\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.042764 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.042764    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-tmp\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.044111 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.044093    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:56:38.044192 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.044122    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 11:56:38.044244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.044185    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-v7zjz\""
Nov 03 11:56:38.058800 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.058772    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"]
Nov 03 11:56:38.089082 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.089052    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.104175 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.104150    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"]
Nov 03 11:56:38.137829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.137803    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 11:56:38.137829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.137803    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 11:56:38.138023 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.137875    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-mklvr\""
Nov 03 11:56:38.143348 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143324    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.143474 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143356    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.143529 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143476    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsptw\" (UniqueName: \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.143529 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143518    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b642\" (UniqueName: \"kubernetes.io/projected/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-kube-api-access-4b642\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.143634 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143538    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-tmp\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.143847 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.143827    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-tmp\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.196916 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.196846    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b642\" (UniqueName: \"kubernetes.io/projected/5fe2c752-e52c-4d1f-9005-3dce6f987f0a-kube-api-access-4b642\") pod \"jobset-operator-674cd59dc5-8kbmz\" (UID: \"5fe2c752-e52c-4d1f-9005-3dce6f987f0a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.244517 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.244480    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsptw\" (UniqueName: \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.244677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.244540    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.244677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.244570    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.245010 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.244989    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.245010 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.245000    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.297786 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.297753    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsptw\" (UniqueName: \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.311025 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.310997    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"
Nov 03 11:56:38.462038 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.461947    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:38.687771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.687746    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz"]
Nov 03 11:56:38.689496 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:56:38.689468    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fe2c752_e52c_4d1f_9005_3dce6f987f0a.slice/crio-73eabc683f47228228203575a05febe443264ac1e8f1e1c6684399e38e1bcbad WatchSource:0}: Error finding container 73eabc683f47228228203575a05febe443264ac1e8f1e1c6684399e38e1bcbad: Status 404 returned error can't find the container with id 73eabc683f47228228203575a05febe443264ac1e8f1e1c6684399e38e1bcbad
Nov 03 11:56:38.743565 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.743489    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz" event={"ID":"5fe2c752-e52c-4d1f-9005-3dce6f987f0a","Type":"ContainerStarted","Data":"73eabc683f47228228203575a05febe443264ac1e8f1e1c6684399e38e1bcbad"}
Nov 03 11:56:38.853289 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:38.853263    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"]
Nov 03 11:56:38.855431 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:56:38.855404    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3281a786_dbbc_4565_a76d_cf25ffa35b1c.slice/crio-1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb WatchSource:0}: Error finding container 1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb: Status 404 returned error can't find the container with id 1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb
Nov 03 11:56:39.749845 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:39.749812    5784 generic.go:356] "Generic (PLEG): container finished" podID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerID="6c84f221f73c7358521811ac4e421958a5abb9d180eebcb398711e971b2180e7" exitCode=0
Nov 03 11:56:39.750245 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:39.749878    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk" event={"ID":"3281a786-dbbc-4565-a76d-cf25ffa35b1c","Type":"ContainerDied","Data":"6c84f221f73c7358521811ac4e421958a5abb9d180eebcb398711e971b2180e7"}
Nov 03 11:56:39.750245 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:39.749915    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk" event={"ID":"3281a786-dbbc-4565-a76d-cf25ffa35b1c","Type":"ContainerStarted","Data":"1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb"}
Nov 03 11:56:40.758792 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:40.758755    5784 generic.go:356] "Generic (PLEG): container finished" podID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerID="424813f6907b80c20e75d3ad212947f27f61bfadd5eae4bbae859d394a395613" exitCode=0
Nov 03 11:56:40.759219 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:40.758846    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk" event={"ID":"3281a786-dbbc-4565-a76d-cf25ffa35b1c","Type":"ContainerDied","Data":"424813f6907b80c20e75d3ad212947f27f61bfadd5eae4bbae859d394a395613"}
Nov 03 11:56:41.766282 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:41.766087    5784 generic.go:356] "Generic (PLEG): container finished" podID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerID="d849ef8560cdbb9c7dc7076b1ad0c0f8d19e6b412a1a4cbbf94540412c07c030" exitCode=0
Nov 03 11:56:41.766282 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:41.766135    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk" event={"ID":"3281a786-dbbc-4565-a76d-cf25ffa35b1c","Type":"ContainerDied","Data":"d849ef8560cdbb9c7dc7076b1ad0c0f8d19e6b412a1a4cbbf94540412c07c030"}
Nov 03 11:56:42.371813 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.371774    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:42.772981 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.772934    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz" event={"ID":"5fe2c752-e52c-4d1f-9005-3dce6f987f0a","Type":"ContainerStarted","Data":"af4d834d72f99069e1645a4e3b58b02dec7630ddecd469b92595df498c5ef279"}
Nov 03 11:56:42.882186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.882136    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8kbmz" podStartSLOduration=2.845387123 podStartE2EDuration="5.882121856s" totalImagesPullingTime="3.036734733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:42.880226873 +0000 UTC m=+309.322429199" watchObservedRunningTime="2026-11-03 11:56:42.882121856 +0000 UTC m=+309.324324161"
Nov 03 11:56:42.897481 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.897459    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:42.982448 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.982418    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\" (UniqueName: \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\") pod \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") "
Nov 03 11:56:42.982618 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.982467    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\") pod \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") "
Nov 03 11:56:42.982618 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.982537    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\") pod \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\" (UID: \"3281a786-dbbc-4565-a76d-cf25ffa35b1c\") "
Nov 03 11:56:42.983197 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.983172    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle" pod "3281a786-dbbc-4565-a76d-cf25ffa35b1c" (UID: "3281a786-dbbc-4565-a76d-cf25ffa35b1c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:42.984572 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.984543    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw" pod "3281a786-dbbc-4565-a76d-cf25ffa35b1c" (UID: "3281a786-dbbc-4565-a76d-cf25ffa35b1c"). InnerVolumeSpecName "kube-api-access-gsptw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:56:42.988677 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:42.988642    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util" pod "3281a786-dbbc-4565-a76d-cf25ffa35b1c" (UID: "3281a786-dbbc-4565-a76d-cf25ffa35b1c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:56:43.083530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.083441    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsptw\" (UniqueName: \"kubernetes.io/projected/3281a786-dbbc-4565-a76d-cf25ffa35b1c-kube-api-access-gsptw\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:43.083530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.083474    5784 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-bundle\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:43.083530 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.083484    5784 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3281a786-dbbc-4565-a76d-cf25ffa35b1c-util\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:56:43.534288 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.534249    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:43.703479 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.703451    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:43.779606 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.779574    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk" event={"ID":"3281a786-dbbc-4565-a76d-cf25ffa35b1c","Type":"ContainerDied","Data":"1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb"}
Nov 03 11:56:43.779606 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.779607    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c96pmk"
Nov 03 11:56:43.780114 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:43.779624    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a6cd0cd455712f0a8dcf33b4d5ff632222f4c63eb7b8dad35895a5f6324dccb"
Nov 03 11:56:44.490757 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.490722    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:44.667012 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.666981    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"]
Nov 03 11:56:44.667318 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667301    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="pull"
Nov 03 11:56:44.667360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667318    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="pull"
Nov 03 11:56:44.667360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667334    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="extract"
Nov 03 11:56:44.667360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667340    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="extract"
Nov 03 11:56:44.667360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667353    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="util"
Nov 03 11:56:44.667360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667359    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="util"
Nov 03 11:56:44.667535 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.667432    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3281a786-dbbc-4565-a76d-cf25ffa35b1c" containerName="extract"
Nov 03 11:56:44.670649 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.670627    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.694957 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.694932    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.695104 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.694985    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.695104 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.695076    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.695219 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.695107    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.695219 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.695160    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.695297 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.695237    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k5m\" (UniqueName: \"kubernetes.io/projected/5bf43715-5f18-477f-892a-5e67b4bfe950-kube-api-access-n2k5m\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.721789 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.721762    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 11:56:44.721936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.721802    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 11:56:44.721936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.721846    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 11:56:44.722247 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.722228    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 11:56:44.722247 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.722247    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 11:56:44.751794 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.751703    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:44.768057 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.768031    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zcjpl\""
Nov 03 11:56:44.770388 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.770359    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"]
Nov 03 11:56:44.796083 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796055    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796106    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796130    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k5m\" (UniqueName: \"kubernetes.io/projected/5bf43715-5f18-477f-892a-5e67b4bfe950-kube-api-access-n2k5m\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796179    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796209    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796477 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796236    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.796743 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.796561    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-manifests\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.798472 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.798450    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5bf43715-5f18-477f-892a-5e67b4bfe950-tmp\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.798752 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.798733    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.798831 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.798751    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-metrics-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.798831 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.798760    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5bf43715-5f18-477f-892a-5e67b4bfe950-webhook-cert\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.850939 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.850904    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k5m\" (UniqueName: \"kubernetes.io/projected/5bf43715-5f18-477f-892a-5e67b4bfe950-kube-api-access-n2k5m\") pod \"opendatahub-operator-controller-manager-5dd964fc4d-8v5n7\" (UID: \"5bf43715-5f18-477f-892a-5e67b4bfe950\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:44.932754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.932724    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:44.985264 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:44.985234    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:45.098628 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:45.098590    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:45.451072 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:45.451038    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"]
Nov 03 11:56:45.451771 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:56:45.451747    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf43715_5f18_477f_892a_5e67b4bfe950.slice/crio-ac6cfa4df249c0bbc7814650f7e569c9e6003a82313d78800aec853241002129 WatchSource:0}: Error finding container ac6cfa4df249c0bbc7814650f7e569c9e6003a82313d78800aec853241002129: Status 404 returned error can't find the container with id ac6cfa4df249c0bbc7814650f7e569c9e6003a82313d78800aec853241002129
Nov 03 11:56:45.791026 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:45.790943    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7" event={"ID":"5bf43715-5f18-477f-892a-5e67b4bfe950","Type":"ContainerStarted","Data":"ac6cfa4df249c0bbc7814650f7e569c9e6003a82313d78800aec853241002129"}
Nov 03 11:56:47.902952 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:47.902923    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:48.530542 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:48.530489    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:49.101015 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:49.100979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:50.297727 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:50.297695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:52.827141 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:52.827107    5784 generic.go:356] "Generic (PLEG): container finished" podID="5bf43715-5f18-477f-892a-5e67b4bfe950" containerID="e8f3aa74ae689fe28b6424b8e8d854c27c42f8ad4bb53b77656c5ddf4facb2d1" exitCode=0
Nov 03 11:56:52.827644 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:52.827201    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7" event={"ID":"5bf43715-5f18-477f-892a-5e67b4bfe950","Type":"ContainerDied","Data":"e8f3aa74ae689fe28b6424b8e8d854c27c42f8ad4bb53b77656c5ddf4facb2d1"}
Nov 03 11:56:53.292606 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:53.292570    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:53.834212 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:53.834166    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7" event={"ID":"5bf43715-5f18-477f-892a-5e67b4bfe950","Type":"ContainerStarted","Data":"414ba0625421aa9476952530380c805a08bb70d1a825c471f8ddcec14d220606"}
Nov 03 11:56:53.834601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:53.834346    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:56:54.047218 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:54.047118    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7" podStartSLOduration=2.392910228 podStartE2EDuration="10.047102496s" totalImagesPullingTime="7.218136266s" totalInitContainerRuntime="436.056002ms" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:56:54.001061357 +0000 UTC m=+320.443263699" watchObservedRunningTime="2026-11-03 11:56:54.047102496 +0000 UTC m=+320.489304830"
Nov 03 11:56:55.749118 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:55.749073    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:56.286920 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.286888    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:56:56.290438 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.290422    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.340949 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.340919    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:56:56.384863 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.384824    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 11:56:56.384863 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.384847    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 11:56:56.385057 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.384923    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lqflp\""
Nov 03 11:56:56.398238 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.398210    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.398361 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.398244    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxls\" (UniqueName: \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.470717 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:56:56.470683    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:56:56.499003 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.498973    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.499153 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.499015    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxls\" (UniqueName: \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.499400 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.499346    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.551918 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.551845    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxls\" (UniqueName: \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-fnsks\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:56.602952 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:56.602921    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:56:57.058282 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.058251    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"]
Nov 03 11:56:57.062761 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.062740    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.104553 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.104532    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/675e98b0-a45f-40c8-a944-85642966ddf3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.104685 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.104659    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmhk\" (UniqueName: \"kubernetes.io/projected/675e98b0-a45f-40c8-a944-85642966ddf3-kube-api-access-khmhk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.104855 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.104837    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:56:57.106820 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:56:57.106794    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b1ad36_e3f6_408b_9cf3_61b8b8a4745b.slice/crio-aab33093a51fb6284f8aed7eccab6bc3581b6132573e3ecc98d6e31291ae17c2 WatchSource:0}: Error finding container aab33093a51fb6284f8aed7eccab6bc3581b6132573e3ecc98d6e31291ae17c2: Status 404 returned error can't find the container with id aab33093a51fb6284f8aed7eccab6bc3581b6132573e3ecc98d6e31291ae17c2
Nov 03 11:56:57.148117 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.148090    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 11:56:57.148284 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.148174    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 11:56:57.148284 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.148245    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-wrd8c\""
Nov 03 11:56:57.150405 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.150368    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"]
Nov 03 11:56:57.205311 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.205272    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/675e98b0-a45f-40c8-a944-85642966ddf3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.205510 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.205365    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmhk\" (UniqueName: \"kubernetes.io/projected/675e98b0-a45f-40c8-a944-85642966ddf3-kube-api-access-khmhk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.205771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.205752    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/675e98b0-a45f-40c8-a944-85642966ddf3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.258973 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.258943    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmhk\" (UniqueName: \"kubernetes.io/projected/675e98b0-a45f-40c8-a944-85642966ddf3-kube-api-access-khmhk\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-496hk\" (UID: \"675e98b0-a45f-40c8-a944-85642966ddf3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.375246 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.375164    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"
Nov 03 11:56:57.698748 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.698715    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:57.785362 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.785334    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk"]
Nov 03 11:56:57.787832 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:56:57.787807    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675e98b0_a45f_40c8_a944_85642966ddf3.slice/crio-7bbd98dac8299d1e4be7c24e0e39406ff165a385d2f517aa2d529d477016cfd1 WatchSource:0}: Error finding container 7bbd98dac8299d1e4be7c24e0e39406ff165a385d2f517aa2d529d477016cfd1: Status 404 returned error can't find the container with id 7bbd98dac8299d1e4be7c24e0e39406ff165a385d2f517aa2d529d477016cfd1
Nov 03 11:56:57.864128 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.864092    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk" event={"ID":"675e98b0-a45f-40c8-a944-85642966ddf3","Type":"ContainerStarted","Data":"7bbd98dac8299d1e4be7c24e0e39406ff165a385d2f517aa2d529d477016cfd1"}
Nov 03 11:56:57.865349 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:57.865328    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" event={"ID":"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b","Type":"ContainerStarted","Data":"aab33093a51fb6284f8aed7eccab6bc3581b6132573e3ecc98d6e31291ae17c2"}
Nov 03 11:56:58.279111 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:58.279081    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:58.453605 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:58.453572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:56:59.521474 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:56:59.521438    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:00.143867 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.142061    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"]
Nov 03 11:57:00.147659 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.147628    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:00.235988 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.235945    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkrs\" (UniqueName: \"kubernetes.io/projected/f993bb43-355f-439b-9add-9ec55f05ed03-kube-api-access-svkrs\") pod \"limitador-operator-controller-manager-5778bb4449-pw5vh\" (UID: \"f993bb43-355f-439b-9add-9ec55f05ed03\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:00.245316 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.245287    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2b4fc\""
Nov 03 11:57:00.247815 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.247789    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"]
Nov 03 11:57:00.336870 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.336823    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkrs\" (UniqueName: \"kubernetes.io/projected/f993bb43-355f-439b-9add-9ec55f05ed03-kube-api-access-svkrs\") pod \"limitador-operator-controller-manager-5778bb4449-pw5vh\" (UID: \"f993bb43-355f-439b-9add-9ec55f05ed03\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:00.393479 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.393434    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkrs\" (UniqueName: \"kubernetes.io/projected/f993bb43-355f-439b-9add-9ec55f05ed03-kube-api-access-svkrs\") pod \"limitador-operator-controller-manager-5778bb4449-pw5vh\" (UID: \"f993bb43-355f-439b-9add-9ec55f05ed03\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:00.465502 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.465405    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:00.836858 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.836823    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"]
Nov 03 11:57:00.886967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.886935    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:00.890124 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.890088    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk" event={"ID":"675e98b0-a45f-40c8-a944-85642966ddf3","Type":"ContainerStarted","Data":"cb8e27ccc9e10d1850906a889bf36ef1cf6dccb4c1c358ce1df631454f212c9c"}
Nov 03 11:57:00.891788 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:00.891764    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh" event={"ID":"f993bb43-355f-439b-9add-9ec55f05ed03","Type":"ContainerStarted","Data":"7eb3961f0cdfa01dc03a1b501fc567b61281c8c2f462d446c0d1859716e50c1b"}
Nov 03 11:57:01.000958 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:01.000854    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-496hk" podStartSLOduration=2.087482717 podStartE2EDuration="5.000841634s" totalImagesPullingTime="2.913358917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:00.99953957 +0000 UTC m=+327.441741875" watchObservedRunningTime="2026-11-03 11:57:01.000841634 +0000 UTC m=+327.443043940"
Nov 03 11:57:01.171012 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:01.170975    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:01.359324 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:01.359241    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:01.601134 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:01.601104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:03.693113 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:03.693086    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:03.851107 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:03.851072    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:03.910818 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:03.910768    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" event={"ID":"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b","Type":"ContainerStarted","Data":"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"}
Nov 03 11:57:03.910982 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:03.910837    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:57:04.114893 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.114783    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" podStartSLOduration=2.332108468 podStartE2EDuration="8.114763359s" totalImagesPullingTime="5.782654891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:04.066474601 +0000 UTC m=+330.508676923" watchObservedRunningTime="2026-11-03 11:57:04.114763359 +0000 UTC m=+330.556965663"
Nov 03 11:57:04.538320 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.538283    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"]
Nov 03 11:57:04.541858 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.541838    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:04.554561 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.554518    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:04.576792 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.576764    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxmk\" (UniqueName: \"kubernetes.io/projected/f248db66-6a89-4874-92c0-d448e2380828-kube-api-access-dxxmk\") pod \"authorino-operator-7b7bb4bc77-m9n58\" (UID: \"f248db66-6a89-4874-92c0-d448e2380828\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:04.617942 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.614894    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nbg6p\""
Nov 03 11:57:04.618239 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.618218    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"]
Nov 03 11:57:04.677880 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.677854    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxmk\" (UniqueName: \"kubernetes.io/projected/f248db66-6a89-4874-92c0-d448e2380828-kube-api-access-dxxmk\") pod \"authorino-operator-7b7bb4bc77-m9n58\" (UID: \"f248db66-6a89-4874-92c0-d448e2380828\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:04.734217 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.734188    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxmk\" (UniqueName: \"kubernetes.io/projected/f248db66-6a89-4874-92c0-d448e2380828-kube-api-access-dxxmk\") pod \"authorino-operator-7b7bb4bc77-m9n58\" (UID: \"f248db66-6a89-4874-92c0-d448e2380828\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:04.842328 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.842250    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5dd964fc4d-8v5n7"
Nov 03 11:57:04.856990 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.856966    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:04.926852 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.926681    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh" event={"ID":"f993bb43-355f-439b-9add-9ec55f05ed03","Type":"ContainerStarted","Data":"6748dddbcf22e4baffe3ac05b2ff8eee21c2cd3f5098599a05e149f3ebdb0d3e"}
Nov 03 11:57:04.926852 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:04.926740    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:05.093496 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:05.091121    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh" podStartSLOduration=1.336227593 podStartE2EDuration="5.091107003s" totalImagesPullingTime="3.75487941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:05.090708318 +0000 UTC m=+331.532910628" watchObservedRunningTime="2026-11-03 11:57:05.091107003 +0000 UTC m=+331.533309308"
Nov 03 11:57:05.228125 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:05.228095    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"]
Nov 03 11:57:05.229899 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:05.229873    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf248db66_6a89_4874_92c0_d448e2380828.slice/crio-ff7effaea4d0c32a7fbbf816d0be03a7c176c6bfd7eeaaa3b156b7f48cf1b6d7 WatchSource:0}: Error finding container ff7effaea4d0c32a7fbbf816d0be03a7c176c6bfd7eeaaa3b156b7f48cf1b6d7: Status 404 returned error can't find the container with id ff7effaea4d0c32a7fbbf816d0be03a7c176c6bfd7eeaaa3b156b7f48cf1b6d7
Nov 03 11:57:05.495755 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:05.495725    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:05.902137 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:05.902104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:05.933094 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:05.933056    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58" event={"ID":"f248db66-6a89-4874-92c0-d448e2380828","Type":"ContainerStarted","Data":"ff7effaea4d0c32a7fbbf816d0be03a7c176c6bfd7eeaaa3b156b7f48cf1b6d7"}
Nov 03 11:57:06.364762 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:06.364682    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:06.539419 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:06.539371    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:07.962101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:07.962066    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"]
Nov 03 11:57:07.964911 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:07.964889    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:08.008013 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.007971    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvhx\" (UniqueName: \"kubernetes.io/projected/7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8-kube-api-access-gcvhx\") pod \"dns-operator-controller-manager-8557cbdb84-kpq2r\" (UID: \"7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:08.060754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.060703    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 11:57:08.061186 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.061139    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b287c\""
Nov 03 11:57:08.063897 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.063872    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"]
Nov 03 11:57:08.108913 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.108888    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvhx\" (UniqueName: \"kubernetes.io/projected/7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8-kube-api-access-gcvhx\") pod \"dns-operator-controller-manager-8557cbdb84-kpq2r\" (UID: \"7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:08.165576 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.165550    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvhx\" (UniqueName: \"kubernetes.io/projected/7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8-kube-api-access-gcvhx\") pod \"dns-operator-controller-manager-8557cbdb84-kpq2r\" (UID: \"7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:08.280537 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.280451    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:08.322234 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.322204    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:08.557466 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.557394    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:08.660500 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.660474    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"]
Nov 03 11:57:08.662457 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:08.662425    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d485060_1b5d_41c9_8b5c_ba2d4da0bfa8.slice/crio-972c8fdc407e1cd612b29232291bdfebbb28e2c5e464aa02386a38c6ea47d0c0 WatchSource:0}: Error finding container 972c8fdc407e1cd612b29232291bdfebbb28e2c5e464aa02386a38c6ea47d0c0: Status 404 returned error can't find the container with id 972c8fdc407e1cd612b29232291bdfebbb28e2c5e464aa02386a38c6ea47d0c0
Nov 03 11:57:08.951797 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.951757    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r" event={"ID":"7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8","Type":"ContainerStarted","Data":"972c8fdc407e1cd612b29232291bdfebbb28e2c5e464aa02386a38c6ea47d0c0"}
Nov 03 11:57:08.953478 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.953455    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58" event={"ID":"f248db66-6a89-4874-92c0-d448e2380828","Type":"ContainerStarted","Data":"d88e516aa8a35678fa1e642c317ec120610a104d1f9eda82868cef03710a71e0"}
Nov 03 11:57:08.953584 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:08.953568    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:09.062219 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:09.062157    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58" podStartSLOduration=2.214407758 podStartE2EDuration="5.062140383s" totalImagesPullingTime="2.847732625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:09.060798555 +0000 UTC m=+335.503000860" watchObservedRunningTime="2026-11-03 11:57:09.062140383 +0000 UTC m=+335.504342688"
Nov 03 11:57:11.499913 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:11.499873    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:11.658511 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:11.658479    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:11.971646 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:11.971609    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r" event={"ID":"7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8","Type":"ContainerStarted","Data":"d3423db4d43ea801f4d1eac7af22271151b0a28e174474181b91097785ccdb63"}
Nov 03 11:57:11.971830 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:11.971769    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:12.094856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:12.094821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:12.109676 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:12.109627    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r" podStartSLOduration=2.127407632 podStartE2EDuration="5.109612024s" totalImagesPullingTime="2.982204392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:12.108007055 +0000 UTC m=+338.550209386" watchObservedRunningTime="2026-11-03 11:57:12.109612024 +0000 UTC m=+338.551814328"
Nov 03 11:57:12.247673 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:12.247597    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:12.684204 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:12.684162    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:12.843833 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:12.843802    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:14.245518 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:14.245482    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:14.928747 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:14.928714    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:57:15.351686 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.351606    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 11:57:15.355440 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.355420    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.415235 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.415206    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 11:57:15.490061 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.490031    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.490222 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.490095    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrmk\" (UniqueName: \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.591046 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.591007    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrmk\" (UniqueName: \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.591218 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.591070    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.591460 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.591444    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.644922 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.644890    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrmk\" (UniqueName: \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\") pod \"kuadrant-operator-controller-manager-678d87766f-db85w\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.668731 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.668702    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:15.935238 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:15.935157    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-pw5vh"
Nov 03 11:57:16.129121 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:16.129099    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 11:57:16.131501 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:16.131471    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7822d282_6299_48b9_8dc4_df94f622047c.slice/crio-b41b8a137bc54936a3972286502de4a9fca6034b35c23cdfd4f56bd94cdd149a WatchSource:0}: Error finding container b41b8a137bc54936a3972286502de4a9fca6034b35c23cdfd4f56bd94cdd149a: Status 404 returned error can't find the container with id b41b8a137bc54936a3972286502de4a9fca6034b35c23cdfd4f56bd94cdd149a
Nov 03 11:57:17.007036 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:17.006992    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" event={"ID":"7822d282-6299-48b9-8dc4-df94f622047c","Type":"ContainerStarted","Data":"af1cab7bc8cd374d9ff5b5567e30baad1fe9df860286f44091f811cde0ab764e"}
Nov 03 11:57:17.007552 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:17.007065    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" event={"ID":"7822d282-6299-48b9-8dc4-df94f622047c","Type":"ContainerStarted","Data":"b41b8a137bc54936a3972286502de4a9fca6034b35c23cdfd4f56bd94cdd149a"}
Nov 03 11:57:17.007552 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:17.007242    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:17.210302 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:17.210241    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" podStartSLOduration=2.210222863 podStartE2EDuration="2.210222863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:17.162939036 +0000 UTC m=+343.605141340" watchObservedRunningTime="2026-11-03 11:57:17.210222863 +0000 UTC m=+343.652425170"
Nov 03 11:57:19.567770 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:19.567732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:19.725035 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:19.724998    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:19.960595 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:19.960565    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-m9n58"
Nov 03 11:57:20.114027 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:20.113994    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:20.282729 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:20.282644    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:21.283014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.282976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:21.454775 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.454737    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:21.789613 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.789578    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"]
Nov 03 11:57:21.797957 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.797935    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.864586 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.864551    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"]
Nov 03 11:57:21.895107 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.895079    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mmk5d\""
Nov 03 11:57:21.911918 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.911872    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:21.951129 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951099    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951323 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951136    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951323 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951241    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951444 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951416    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951519 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951500    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8grx\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-kube-api-access-n8grx\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951718 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951699    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951730    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951761    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:21.951866 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:21.951806    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.052735 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052638    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.052735 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052677    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8grx\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-kube-api-access-n8grx\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052743    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052760    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052787    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052812    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052842    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052873    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.052914    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.053171    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.053506    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.053578    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.053619    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.054538 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.054280    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.055672 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.055654    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.055899 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.055881    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.107526 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.107499    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.107678 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.107593    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8grx\" (UniqueName: \"kubernetes.io/projected/a1f5b1de-4ae6-44aa-b7e4-00b3120310e5-kube-api-access-n8grx\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x\" (UID: \"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.110322 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.110302    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:22.113882 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.113857    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:22.269404 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.269347    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:22.523853 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.523823    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"]
Nov 03 11:57:22.525887 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:22.525846    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f5b1de_4ae6_44aa_b7e4_00b3120310e5.slice/crio-1149b4b9f94c0258d588ba6593f434a35bdaf4715124342a2f5ee4881d47752a WatchSource:0}: Error finding container 1149b4b9f94c0258d588ba6593f434a35bdaf4715124342a2f5ee4881d47752a: Status 404 returned error can't find the container with id 1149b4b9f94c0258d588ba6593f434a35bdaf4715124342a2f5ee4881d47752a
Nov 03 11:57:22.528452 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.528420    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:57:22.528563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.528489    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:57:22.528563 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.528522    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:57:22.979727 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:22.979700    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-kpq2r"
Nov 03 11:57:23.041014 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:23.040975    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x" event={"ID":"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5","Type":"ContainerStarted","Data":"990f339eefb0ff8507252f1476b5fa1c66a7e4b80b990497d768ea973edda2a6"}
Nov 03 11:57:23.041183 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:23.041033    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x" event={"ID":"a1f5b1de-4ae6-44aa-b7e4-00b3120310e5","Type":"ContainerStarted","Data":"1149b4b9f94c0258d588ba6593f434a35bdaf4715124342a2f5ee4881d47752a"}
Nov 03 11:57:23.111459 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:23.111426    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:23.117733 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:23.117675    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:23.213370 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:23.213283    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x" podStartSLOduration=2.213269305 podStartE2EDuration="2.213269305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:23.212402564 +0000 UTC m=+349.654604883" watchObservedRunningTime="2026-11-03 11:57:23.213269305 +0000 UTC m=+349.655471691"
Nov 03 11:57:24.046436 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:24.046406    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:24.047470 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:24.047449    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x"
Nov 03 11:57:24.216332 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:24.216288    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:24.398503 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:24.398473    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:26.110890 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:26.110853    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:26.263292 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:26.263253    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:26.472913 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:57:26.472869    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:57:26.500388 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:26.500345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:26.668531 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:26.668491    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:27.159190 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.159158    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:27.164068 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.164043    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:27.216688 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.216649    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:27.260331 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.260302    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v48m5\""
Nov 03 11:57:27.299284 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.299253    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkjl\" (UniqueName: \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\") pod \"authorino-57f7f8659f-nz2wc\" (UID: \"1ef79b1e-a6d4-4192-92e7-c2377996fa9b\") " pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:27.340811 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.340775    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:27.400733 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.400698    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkjl\" (UniqueName: \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\") pod \"authorino-57f7f8659f-nz2wc\" (UID: \"1ef79b1e-a6d4-4192-92e7-c2377996fa9b\") " pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:27.455105 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.455028    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkjl\" (UniqueName: \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\") pod \"authorino-57f7f8659f-nz2wc\" (UID: \"1ef79b1e-a6d4-4192-92e7-c2377996fa9b\") " pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:27.474851 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.474822    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:27.521276 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.521247    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:27.835544 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.835520    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:27.837599 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:27.837571    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ef79b1e_a6d4_4192_92e7_c2377996fa9b.slice/crio-81598fa05fe10a75ea607ff7423183e5ecd43489376a951edcbf20ea22018eda WatchSource:0}: Error finding container 81598fa05fe10a75ea607ff7423183e5ecd43489376a951edcbf20ea22018eda: Status 404 returned error can't find the container with id 81598fa05fe10a75ea607ff7423183e5ecd43489376a951edcbf20ea22018eda
Nov 03 11:57:27.838912 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:27.838893    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 11:57:28.014817 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:28.014781    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 11:57:28.069325 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:28.069287    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" event={"ID":"1ef79b1e-a6d4-4192-92e7-c2377996fa9b","Type":"ContainerStarted","Data":"81598fa05fe10a75ea607ff7423183e5ecd43489376a951edcbf20ea22018eda"}
Nov 03 11:57:28.225789 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:28.225753    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:57:28.226255 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:28.226006    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" podUID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" containerName="manager" containerID="cri-o://6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5" gracePeriod=10
Nov 03 11:57:28.837340 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:28.837294    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:29.375613 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.375590    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:57:29.518133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.518046    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\") pod \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") "
Nov 03 11:57:29.518133 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.518103    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\" (UniqueName: \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\") pod \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\" (UID: \"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b\") "
Nov 03 11:57:29.518548 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.518520    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume" pod "28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" (UID: "28b1ad36-e3f6-408b-9cf3-61b8b8a4745b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 11:57:29.520242 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.520223    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls" pod "28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" (UID: "28b1ad36-e3f6-408b-9cf3-61b8b8a4745b"). InnerVolumeSpecName "kube-api-access-lrxls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:57:29.619013 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.618982    5784 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-extensions-socket-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:57:29.619013 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:29.619011    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrxls\" (UniqueName: \"kubernetes.io/projected/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b-kube-api-access-lrxls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:57:30.081644 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.081605    5784 generic.go:356] "Generic (PLEG): container finished" podID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" containerID="6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5" exitCode=0
Nov 03 11:57:30.081820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.081661    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"
Nov 03 11:57:30.081820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.081673    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" event={"ID":"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b","Type":"ContainerDied","Data":"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"}
Nov 03 11:57:30.081820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.081726    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks" event={"ID":"28b1ad36-e3f6-408b-9cf3-61b8b8a4745b","Type":"ContainerDied","Data":"aab33093a51fb6284f8aed7eccab6bc3581b6132573e3ecc98d6e31291ae17c2"}
Nov 03 11:57:30.081820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.081757    5784 scope.go:122] "RemoveContainer" containerID="6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"
Nov 03 11:57:30.090601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.090583    5784 scope.go:122] "RemoveContainer" containerID="6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"
Nov 03 11:57:30.090860 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:57:30.090844    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5\": container with ID starting with 6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5 not found: ID does not exist" containerID="6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"
Nov 03 11:57:30.090913 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.090870    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5"} err="failed to get container status \"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5\": rpc error: code = NotFound desc = could not find container \"6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5\": container with ID starting with 6a49869eb84af03c6487e911386fbc8e0679b4cda9b0f81c2400dda87011f8a5 not found: ID does not exist"
Nov 03 11:57:30.280778 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.280745    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:57:30.327020 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:30.326989    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-fnsks"]
Nov 03 11:57:31.679935 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:31.679904    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" path="/var/lib/kubelet/pods/28b1ad36-e3f6-408b-9cf3-61b8b8a4745b/volumes"
Nov 03 11:57:33.312499 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:33.312450    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:33.476911 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:33.476876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:33.711345 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:33.711314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:33.870114 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:33.870080    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:34.181257 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:34.181223    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:34.332057 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:34.332024    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:36.121434 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:36.121365    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" event={"ID":"1ef79b1e-a6d4-4192-92e7-c2377996fa9b","Type":"ContainerStarted","Data":"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"}
Nov 03 11:57:36.228662 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:36.228603    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" podStartSLOduration=1.59960403 podStartE2EDuration="9.228587428s" totalImagesPullingTime="7.628983398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:36.226586922 +0000 UTC m=+362.668789226" watchObservedRunningTime="2026-11-03 11:57:36.228587428 +0000 UTC m=+362.670789732"
Nov 03 11:57:40.767936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:40.767897    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:40.921489 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:40.921453    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:42.694911 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:42.694876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:45.515072 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:45.515036    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:45.681104 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:45.681069    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:46.309151 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.309118    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"]
Nov 03 11:57:46.309542 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.309529    5784 cpu_manager.go:414] "containerMap: removing container" podUID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" containerName="manager"
Nov 03 11:57:46.309593 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.309543    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" containerName="manager"
Nov 03 11:57:46.309634 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.309624    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b1ad36-e3f6-408b-9cf3-61b8b8a4745b" containerName="manager"
Nov 03 11:57:46.313245 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.313224    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.361812 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.361770    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 11:57:46.373219 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.373189    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.373365 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.373226    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce713df0-9f1c-4581-8971-6dbc92a58b67-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.373365 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.373263    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/ce713df0-9f1c-4581-8971-6dbc92a58b67-kube-api-access-rkmc5\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.373482 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.373440    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.407798 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.407767    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 11:57:46.408033 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.408010    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-sl524\""
Nov 03 11:57:46.473959 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.473920    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.473959 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.473958    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.474232 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.473980    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce713df0-9f1c-4581-8971-6dbc92a58b67-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.474232 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.474009    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/ce713df0-9f1c-4581-8971-6dbc92a58b67-kube-api-access-rkmc5\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.474419 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.474394    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.474496 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.474471    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce713df0-9f1c-4581-8971-6dbc92a58b67-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.500721 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.500682    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 03 11:57:46.502999 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.502977    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"]
Nov 03 11:57:46.505453 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.505430    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ce713df0-9f1c-4581-8971-6dbc92a58b67-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.527170 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.527145    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkmc5\" (UniqueName: \"kubernetes.io/projected/ce713df0-9f1c-4581-8971-6dbc92a58b67-kube-api-access-rkmc5\") pod \"kserve-module-controller-manager-7566cf5fc8-cc94k\" (UID: \"ce713df0-9f1c-4581-8971-6dbc92a58b67\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:46.626827 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:46.626799    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:47.039771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:47.039748    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"]
Nov 03 11:57:47.041392 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:47.041350    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce713df0_9f1c_4581_8971_6dbc92a58b67.slice/crio-fb929d617d04c7242fee60e52c173dd5875b455a30968234475ed156fea77d9a WatchSource:0}: Error finding container fb929d617d04c7242fee60e52c173dd5875b455a30968234475ed156fea77d9a: Status 404 returned error can't find the container with id fb929d617d04c7242fee60e52c173dd5875b455a30968234475ed156fea77d9a
Nov 03 11:57:47.184273 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:47.184240    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k" event={"ID":"ce713df0-9f1c-4581-8971-6dbc92a58b67","Type":"ContainerStarted","Data":"fb929d617d04c7242fee60e52c173dd5875b455a30968234475ed156fea77d9a"}
Nov 03 11:57:48.057313 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:48.057281    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:48.210183 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:48.210151    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:51.997686 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:51.997643    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:52.180417 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:52.180365    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:53.204596 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.204558    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:57:53.208468 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.208448    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:57:53.228667 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.228633    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k" event={"ID":"ce713df0-9f1c-4581-8971-6dbc92a58b67","Type":"ContainerStarted","Data":"09ab04177c2fddf194bc561b50abc5c39ce48a53372f04ab8f8d899db757aca4"}
Nov 03 11:57:53.228831 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.228694    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:57:53.260902 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.260874    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:57:53.339674 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.339640    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d89rh\" (UniqueName: \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\") pod \"authorino-56c6c9ddd5-xz86f\" (UID: \"8c72d989-d9d6-4780-99fd-4388997af0e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:57:53.424499 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.424444    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k" podStartSLOduration=1.8913399279999998 podStartE2EDuration="7.424430131s" totalImagesPullingTime="5.533090203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:53.421609769 +0000 UTC m=+379.863812070" watchObservedRunningTime="2026-11-03 11:57:53.424430131 +0000 UTC m=+379.866632436"
Nov 03 11:57:53.440446 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.440408    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d89rh\" (UniqueName: \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\") pod \"authorino-56c6c9ddd5-xz86f\" (UID: \"8c72d989-d9d6-4780-99fd-4388997af0e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:57:53.497286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.497211    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d89rh\" (UniqueName: \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\") pod \"authorino-56c6c9ddd5-xz86f\" (UID: \"8c72d989-d9d6-4780-99fd-4388997af0e6\") " pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:57:53.520037 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.520009    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:57:53.880021 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:53.879990    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:57:53.882244 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:53.882214    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c72d989_d9d6_4780_99fd_4388997af0e6.slice/crio-405ccfde6b4e0c456dbe5e7a395db4ec030acf2b5a95e71e3eb7e3945c7351b9 WatchSource:0}: Error finding container 405ccfde6b4e0c456dbe5e7a395db4ec030acf2b5a95e71e3eb7e3945c7351b9: Status 404 returned error can't find the container with id 405ccfde6b4e0c456dbe5e7a395db4ec030acf2b5a95e71e3eb7e3945c7351b9
Nov 03 11:57:54.236149 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:54.236058    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" event={"ID":"8c72d989-d9d6-4780-99fd-4388997af0e6","Type":"ContainerStarted","Data":"405ccfde6b4e0c456dbe5e7a395db4ec030acf2b5a95e71e3eb7e3945c7351b9"}
Nov 03 11:57:55.243092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.243052    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" event={"ID":"8c72d989-d9d6-4780-99fd-4388997af0e6","Type":"ContainerStarted","Data":"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"}
Nov 03 11:57:55.348771 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.348660    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" podStartSLOduration=1.137564961 podStartE2EDuration="2.348640796s" totalImagesPullingTime="1.211075835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:57:55.347742627 +0000 UTC m=+381.789944935" watchObservedRunningTime="2026-11-03 11:57:55.348640796 +0000 UTC m=+381.790843100"
Nov 03 11:57:55.374310 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.374280    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:55.374510 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.374488    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" podUID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" containerName="authorino" containerID="cri-o://bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16" gracePeriod=30
Nov 03 11:57:55.629593 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.629568    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:55.761571 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.761541    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\" (UniqueName: \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\") pod \"1ef79b1e-a6d4-4192-92e7-c2377996fa9b\" (UID: \"1ef79b1e-a6d4-4192-92e7-c2377996fa9b\") "
Nov 03 11:57:55.763521 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.763492    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl" pod "1ef79b1e-a6d4-4192-92e7-c2377996fa9b" (UID: "1ef79b1e-a6d4-4192-92e7-c2377996fa9b"). InnerVolumeSpecName "kube-api-access-kwkjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:57:55.863386 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:55.863271    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkjl\" (UniqueName: \"kubernetes.io/projected/1ef79b1e-a6d4-4192-92e7-c2377996fa9b-kube-api-access-kwkjl\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:57:56.249339 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.249300    5784 generic.go:356] "Generic (PLEG): container finished" podID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" containerID="bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16" exitCode=0
Nov 03 11:57:56.249754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.249363    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nz2wc"
Nov 03 11:57:56.249754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.249411    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" event={"ID":"1ef79b1e-a6d4-4192-92e7-c2377996fa9b","Type":"ContainerDied","Data":"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"}
Nov 03 11:57:56.249754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.249465    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nz2wc" event={"ID":"1ef79b1e-a6d4-4192-92e7-c2377996fa9b","Type":"ContainerDied","Data":"81598fa05fe10a75ea607ff7423183e5ecd43489376a951edcbf20ea22018eda"}
Nov 03 11:57:56.249754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.249484    5784 scope.go:122] "RemoveContainer" containerID="bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"
Nov 03 11:57:56.258663 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.258641    5784 scope.go:122] "RemoveContainer" containerID="bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"
Nov 03 11:57:56.258924 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:57:56.258907    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16\": container with ID starting with bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16 not found: ID does not exist" containerID="bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"
Nov 03 11:57:56.258975 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.258930    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16"} err="failed to get container status \"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16\": rpc error: code = NotFound desc = could not find container \"bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16\": container with ID starting with bca24054794576cae3f51ac348f124ba6c1564e0c636193150502e12cac62f16 not found: ID does not exist"
Nov 03 11:57:56.400046 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.400013    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:56.403365 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:56.403334    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nz2wc"]
Nov 03 11:57:56.470508 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:57:56.470474    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:57:57.384461 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:57.384425    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:57.559701 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:57.559671    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:57.680410 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:57.680326    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" path="/var/lib/kubelet/pods/1ef79b1e-a6d4-4192-92e7-c2377996fa9b/volumes"
Nov 03 11:57:58.065214 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.065133    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"]
Nov 03 11:57:58.065597 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.065584    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" containerName="authorino"
Nov 03 11:57:58.065642 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.065598    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" containerName="authorino"
Nov 03 11:57:58.065679 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.065656    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef79b1e-a6d4-4192-92e7-c2377996fa9b" containerName="authorino"
Nov 03 11:57:58.070296 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.070274    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.121754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.121725    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"]
Nov 03 11:57:58.164811 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.164783    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 11:57:58.164979 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.164892    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 11:57:58.164979 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.164918    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 03 11:57:58.164979 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.164960    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-4wch8\""
Nov 03 11:57:58.179210 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.179160    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:57:58.186323 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.186296    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcw8k\" (UniqueName: \"kubernetes.io/projected/0a15fffd-279c-47b8-9a54-2e60a6ec367a-kube-api-access-pcw8k\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.186467 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.186401    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a15fffd-279c-47b8-9a54-2e60a6ec367a-data\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.287645 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.287610    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcw8k\" (UniqueName: \"kubernetes.io/projected/0a15fffd-279c-47b8-9a54-2e60a6ec367a-kube-api-access-pcw8k\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.287897 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.287662    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a15fffd-279c-47b8-9a54-2e60a6ec367a-data\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.288076 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.288056    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0a15fffd-279c-47b8-9a54-2e60a6ec367a-data\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.341358 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.341284    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcw8k\" (UniqueName: \"kubernetes.io/projected/0a15fffd-279c-47b8-9a54-2e60a6ec367a-kube-api-access-pcw8k\") pod \"postgres-7cdbf6dc44-2xh4z\" (UID: \"0a15fffd-279c-47b8-9a54-2e60a6ec367a\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.382834 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.382805    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:57:58.751028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:58.751004    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"]
Nov 03 11:57:58.753524 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:57:58.753498    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a15fffd_279c_47b8_9a54_2e60a6ec367a.slice/crio-f8eccd30f53de60d94bb3843e87c80a484afc8ce416a8d7295c2a68e3ea72896 WatchSource:0}: Error finding container f8eccd30f53de60d94bb3843e87c80a484afc8ce416a8d7295c2a68e3ea72896: Status 404 returned error can't find the container with id f8eccd30f53de60d94bb3843e87c80a484afc8ce416a8d7295c2a68e3ea72896
Nov 03 11:57:59.268998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:57:59.268960    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z" event={"ID":"0a15fffd-279c-47b8-9a54-2e60a6ec367a","Type":"ContainerStarted","Data":"f8eccd30f53de60d94bb3843e87c80a484afc8ce416a8d7295c2a68e3ea72896"}
Nov 03 11:58:04.691031 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:04.690992    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:04.853501 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:04.853467    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:10.341465 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:10.341424    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z" event={"ID":"0a15fffd-279c-47b8-9a54-2e60a6ec367a","Type":"ContainerStarted","Data":"1ec679acdc4299e5315c176f2708bd5aae6870481ca3258fb52f7f9a53664aac"}
Nov 03 11:58:10.341842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:10.341618    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:58:10.447555 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:10.447354    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z" podStartSLOduration=1.7265687170000001 podStartE2EDuration="12.447334835s" totalImagesPullingTime="10.720766118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:58:10.446656452 +0000 UTC m=+396.888858800" watchObservedRunningTime="2026-11-03 11:58:10.447334835 +0000 UTC m=+396.889537141"
Nov 03 11:58:14.238524 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:14.238490    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-cc94k"
Nov 03 11:58:16.376119 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:16.376091    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2xh4z"
Nov 03 11:58:23.612329 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.612286    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:23.657906 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.657868    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:23.668119 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.668094    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.717103 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.717068    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-htd5n\""
Nov 03 11:58:23.727832 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.727797    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.728011 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.727964    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddjxr\" (UniqueName: \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.728075 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.728031    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.763011 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.762975    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 11:58:23.763011 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.762991    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 11:58:23.763011 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.762993    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 11:58:23.765360 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.765333    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:23.776789 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.776764    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:23.829237 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.829205    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddjxr\" (UniqueName: \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.829429 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.829265    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.829429 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.829298    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.831725 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.831695    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.831842 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.831767    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.882248 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.882215    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddjxr\" (UniqueName: \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\") pod \"maas-controller-664954fcdf-699rf\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") " pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:23.942071 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.942038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:23.986601 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:23.986564    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:24.104302 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.104267    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:24.230174 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.230099    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 11:58:24.234574 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.234549    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.288504 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.288418    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 11:58:24.334629 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.334588    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.334840 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.334650    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2k5v\" (UniqueName: \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.334840 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.334759    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.400787 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.400746    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:24.402038 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:58:24.402012    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef1b4ca_a321_4f92_8314_71eed83da293.slice/crio-164d8ab06cf330a0ba4c2433336a6d97896d599d06a4d7823286765c4ce8bbfd WatchSource:0}: Error finding container 164d8ab06cf330a0ba4c2433336a6d97896d599d06a4d7823286765c4ce8bbfd: Status 404 returned error can't find the container with id 164d8ab06cf330a0ba4c2433336a6d97896d599d06a4d7823286765c4ce8bbfd
Nov 03 11:58:24.426579 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.426540    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664954fcdf-699rf" event={"ID":"3ef1b4ca-a321-4f92-8314-71eed83da293","Type":"ContainerStarted","Data":"164d8ab06cf330a0ba4c2433336a6d97896d599d06a4d7823286765c4ce8bbfd"}
Nov 03 11:58:24.435977 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.435948    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.436095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.436014    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2k5v\" (UniqueName: \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.436095 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.436040    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.438490 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.438461    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.438594 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.438502    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.489244 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.489171    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2k5v\" (UniqueName: \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\") pod \"maas-controller-78c58c9bfd-vfmn7\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") " pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.552225 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.552191    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:24.914882 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:24.914857    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 11:58:24.916909 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:58:24.916882    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c74301_9f8c_4263_baf0_2b9b5583ff4c.slice/crio-4f9f53580502152a7d48dfbff99c4d964d63e552f16850fd40385431d1139375 WatchSource:0}: Error finding container 4f9f53580502152a7d48dfbff99c4d964d63e552f16850fd40385431d1139375: Status 404 returned error can't find the container with id 4f9f53580502152a7d48dfbff99c4d964d63e552f16850fd40385431d1139375
Nov 03 11:58:25.433169 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:25.433130    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerStarted","Data":"4f9f53580502152a7d48dfbff99c4d964d63e552f16850fd40385431d1139375"}
Nov 03 11:58:26.478155 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:26.478049    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:58:34.002696 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.002623    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 11:58:34.428681 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.428641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:34.489617 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.489582    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerStarted","Data":"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"}
Nov 03 11:58:34.489787 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.489664    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:34.491108 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.491086    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664954fcdf-699rf" event={"ID":"3ef1b4ca-a321-4f92-8314-71eed83da293","Type":"ContainerStarted","Data":"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"}
Nov 03 11:58:34.491226 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.491209    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:34.601817 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.601752    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" podStartSLOduration=1.565601539 podStartE2EDuration="10.601736101s" totalImagesPullingTime="9.036134562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:58:34.600732734 +0000 UTC m=+421.042935036" watchObservedRunningTime="2026-11-03 11:58:34.601736101 +0000 UTC m=+421.043938407"
Nov 03 11:58:34.605721 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.605677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:34.710829 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:34.710711    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-664954fcdf-699rf" podStartSLOduration=2.151599487 podStartE2EDuration="11.710698262s" totalImagesPullingTime="9.559098775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:58:34.708665201 +0000 UTC m=+421.150867525" watchObservedRunningTime="2026-11-03 11:58:34.710698262 +0000 UTC m=+421.152900577"
Nov 03 11:58:37.353051 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.353013    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"]
Nov 03 11:58:37.356665 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.356645    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.411187 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.411161    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"]
Nov 03 11:58:37.453998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.453968    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 11:58:37.453998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.453978    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 11:58:37.454183 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.453980    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 11:58:37.454183 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.454046    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-nk885\""
Nov 03 11:58:37.468923 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.468898    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hjgv\" (UniqueName: \"kubernetes.io/projected/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-kube-api-access-5hjgv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.469055 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.468956    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.469119 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.469065    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.469165 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.469121    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.518667 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.518633    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:37.570107 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.570077    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hjgv\" (UniqueName: \"kubernetes.io/projected/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-kube-api-access-5hjgv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.570301 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.570142    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.570301 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.570203    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.570301 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.570237    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.570505 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:37.570362    5784 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 11:58:37.570505 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:37.570361    5784 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 03 11:58:37.570505 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:37.570453    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert podName:4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4 nodeName:}" failed. No retries permitted until 2026-11-03 11:58:38.070435598 +0000 UTC m=+424.512637886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-q6gzj" (UID: "4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4") : secret "usage-logs-tenancy-proxy-tls" not found
Nov 03 11:58:37.570505 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:37.570479    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source podName:4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4 nodeName:}" failed. No retries permitted until 2026-11-03 11:58:38.070465141 +0000 UTC m=+424.512667447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-q6gzj" (UID: "4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4") : configmap "usage-tenancy-proxy-source" not found
Nov 03 11:58:37.570890 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.570868    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.625816 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.625794    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hjgv\" (UniqueName: \"kubernetes.io/projected/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-kube-api-access-5hjgv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:37.807535 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:37.807495    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:38.076088 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.075995    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:38.076088 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.076045    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:38.076964 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.076937    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:38.078506 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.078485    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-q6gzj\" (UID: \"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:38.274740 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.274707    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:58:38.643640 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:38.643535    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"]
Nov 03 11:58:38.646659 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:58:38.646620    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4418bc2e_326b_4c2d_b351_5f9c7fbe1ab4.slice/crio-f8253ecfa448ddc9557be5e7643a399a910de1061109a14adb0aea070c13444d WatchSource:0}: Error finding container f8253ecfa448ddc9557be5e7643a399a910de1061109a14adb0aea070c13444d: Status 404 returned error can't find the container with id f8253ecfa448ddc9557be5e7643a399a910de1061109a14adb0aea070c13444d
Nov 03 11:58:39.518963 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:39.518927    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj" event={"ID":"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4","Type":"ContainerStarted","Data":"f8253ecfa448ddc9557be5e7643a399a910de1061109a14adb0aea070c13444d"}
Nov 03 11:58:43.047370 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:43.047328    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:45.498897 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:45.498861    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 11:58:45.499309 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:45.498955    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:45.658581 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:45.658545    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:45.658806 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:45.658766    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-664954fcdf-699rf" podUID="3ef1b4ca-a321-4f92-8314-71eed83da293" containerName="manager" containerID="cri-o://e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d" gracePeriod=30
Nov 03 11:58:45.900667 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:45.900640    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:46.053400 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.053307    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\") pod \"3ef1b4ca-a321-4f92-8314-71eed83da293\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") "
Nov 03 11:58:46.053562 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.053457    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\" (UniqueName: \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\") pod \"3ef1b4ca-a321-4f92-8314-71eed83da293\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") "
Nov 03 11:58:46.053562 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.053526    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\") pod \"3ef1b4ca-a321-4f92-8314-71eed83da293\" (UID: \"3ef1b4ca-a321-4f92-8314-71eed83da293\") "
Nov 03 11:58:46.055560 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.055528    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs" pod "3ef1b4ca-a321-4f92-8314-71eed83da293" (UID: "3ef1b4ca-a321-4f92-8314-71eed83da293"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:58:46.055684 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.055590    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr" pod "3ef1b4ca-a321-4f92-8314-71eed83da293" (UID: "3ef1b4ca-a321-4f92-8314-71eed83da293"). InnerVolumeSpecName "kube-api-access-ddjxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:58:46.055684 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.055603    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert" pod "3ef1b4ca-a321-4f92-8314-71eed83da293" (UID: "3ef1b4ca-a321-4f92-8314-71eed83da293"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 11:58:46.154856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.154823    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddjxr\" (UniqueName: \"kubernetes.io/projected/3ef1b4ca-a321-4f92-8314-71eed83da293-kube-api-access-ddjxr\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:46.154856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.154852    5784 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-webhook-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:46.154856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.154862    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ef1b4ca-a321-4f92-8314-71eed83da293-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:58:46.566280 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.566247    5784 generic.go:356] "Generic (PLEG): container finished" podID="3ef1b4ca-a321-4f92-8314-71eed83da293" containerID="e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d" exitCode=0
Nov 03 11:58:46.566754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.566294    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-664954fcdf-699rf"
Nov 03 11:58:46.566754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.566303    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664954fcdf-699rf" event={"ID":"3ef1b4ca-a321-4f92-8314-71eed83da293","Type":"ContainerDied","Data":"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"}
Nov 03 11:58:46.566754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.566344    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-664954fcdf-699rf" event={"ID":"3ef1b4ca-a321-4f92-8314-71eed83da293","Type":"ContainerDied","Data":"164d8ab06cf330a0ba4c2433336a6d97896d599d06a4d7823286765c4ce8bbfd"}
Nov 03 11:58:46.566754 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.566365    5784 scope.go:122] "RemoveContainer" containerID="e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"
Nov 03 11:58:46.577356 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.577331    5784 scope.go:122] "RemoveContainer" containerID="e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"
Nov 03 11:58:46.577690 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:46.577672    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d\": container with ID starting with e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d not found: ID does not exist" containerID="e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"
Nov 03 11:58:46.577739 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.577700    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d"} err="failed to get container status \"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d\": rpc error: code = NotFound desc = could not find container \"e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d\": container with ID starting with e15ae1e0d53a02a0c73156fb8ecb53335768a3daed5d6356f87b86e9916b038d not found: ID does not exist"
Nov 03 11:58:46.733713 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.733674    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:46.737704 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:46.737675    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-664954fcdf-699rf"]
Nov 03 11:58:47.679887 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:47.679850    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef1b4ca-a321-4f92-8314-71eed83da293" path="/var/lib/kubelet/pods/3ef1b4ca-a321-4f92-8314-71eed83da293/volumes"
Nov 03 11:58:54.514193 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:54.514154    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:54.713233 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:54.713195    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:54.866669 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:54.866587    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:55.018864 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.018824    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:55.179504 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.179459    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:55.309892 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.309845    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-np6xb"]
Nov 03 11:58:55.310706 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.310679    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3ef1b4ca-a321-4f92-8314-71eed83da293" containerName="manager"
Nov 03 11:58:55.310706 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.310703    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef1b4ca-a321-4f92-8314-71eed83da293" containerName="manager"
Nov 03 11:58:55.310909 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.310891    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef1b4ca-a321-4f92-8314-71eed83da293" containerName="manager"
Nov 03 11:58:55.336540 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.336510    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.339626 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.339593    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:55.367355 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.367316    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-np6xb"]
Nov 03 11:58:55.410189 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.410147    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Nov 03 11:58:55.410189 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.410159    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9r7hv\""
Nov 03 11:58:55.435414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.435300    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 11:58:55.442181 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.442157    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.450156 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.450121    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddk4\" (UniqueName: \"kubernetes.io/projected/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-kube-api-access-xddk4\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.450306 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.450200    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-cert\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.467391 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.467288    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 11:58:55.492048 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.492017    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-fxxhl\""
Nov 03 11:58:55.492222 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.492155    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 03 11:58:55.509200 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.509156    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:55.551678 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.551642    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-cert\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.552075 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.551772    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftt2l\" (UniqueName: \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.552075 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.551811    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddk4\" (UniqueName: \"kubernetes.io/projected/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-kube-api-access-xddk4\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.552075 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.551853    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.554552 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.554530    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-cert\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.608110 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.608081    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddk4\" (UniqueName: \"kubernetes.io/projected/19d9ca6d-3426-4d37-86e9-f7051fc5ce41-kube-api-access-xddk4\") pod \"kserve-controller-manager-776cc66d47-np6xb\" (UID: \"19d9ca6d-3426-4d37-86e9-f7051fc5ce41\") " pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.653286 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.653253    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftt2l\" (UniqueName: \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.653481 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.653297    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.654092 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.654071    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:58:55.655956 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.655926    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.706472 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.706397    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftt2l\" (UniqueName: \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\") pod \"llmisvc-controller-manager-869d5dcc5d-bpnkb\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:55.757883 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:55.757852    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:58:56.032282 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:56.032247    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-np6xb"]
Nov 03 11:58:56.034287 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:58:56.034262    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d9ca6d_3426_4d37_86e9_f7051fc5ce41.slice/crio-399e9b55a1bd555964dec6cc9e6939368aa2ba1be7f05edf7a8aa54a0d54d033 WatchSource:0}: Error finding container 399e9b55a1bd555964dec6cc9e6939368aa2ba1be7f05edf7a8aa54a0d54d033: Status 404 returned error can't find the container with id 399e9b55a1bd555964dec6cc9e6939368aa2ba1be7f05edf7a8aa54a0d54d033
Nov 03 11:58:56.136101 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:56.136069    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 11:58:56.474756 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:58:56.474709    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:58:56.600293 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:56.600251    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:58:56.634633 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:56.634589    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb" event={"ID":"19d9ca6d-3426-4d37-86e9-f7051fc5ce41","Type":"ContainerStarted","Data":"399e9b55a1bd555964dec6cc9e6939368aa2ba1be7f05edf7a8aa54a0d54d033"}
Nov 03 11:58:56.636075 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:58:56.636044    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" event={"ID":"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7","Type":"ContainerStarted","Data":"e89ab693080e85cab9da4f640d542adad8fd9433a71659668b3262aaf469ff11"}
Nov 03 11:59:01.254975 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.254940    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:01.504705 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.504621    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:01.652259 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.652206    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:01.668784 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.668742    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj" event={"ID":"4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4","Type":"ContainerStarted","Data":"e704032aad7a53c40b6ac3d430ebcf6d602f9955a57a720fe39c8aa040aa65d5"}
Nov 03 11:59:01.668982 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.668827    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:59:01.777719 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.777627    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj" podStartSLOduration=2.273587606 podStartE2EDuration="24.77761234s" totalImagesPullingTime="22.504024734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:01.775409953 +0000 UTC m=+448.217612254" watchObservedRunningTime="2026-11-03 11:59:01.77761234 +0000 UTC m=+448.219814645"
Nov 03 11:59:01.818453 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.818418    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:01.970191 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:01.970147    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:02.118209 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:02.118114    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:02.269646 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:02.269608    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:02.417650 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:02.417601    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:03.686058 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.686019    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" event={"ID":"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7","Type":"ContainerStarted","Data":"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"}
Nov 03 11:59:03.686492 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.686095    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:59:03.688022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.687999    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb" event={"ID":"19d9ca6d-3426-4d37-86e9-f7051fc5ce41","Type":"ContainerStarted","Data":"2c9a281bc97393bc2ffa44f982bbb043efacfbd069d168a3d285346fe81abe32"}
Nov 03 11:59:03.688145 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.688081    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:59:03.794510 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.794451    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" podStartSLOduration=1.996463981 podStartE2EDuration="8.794432247s" totalImagesPullingTime="6.797968266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:03.791730941 +0000 UTC m=+450.233933265" watchObservedRunningTime="2026-11-03 11:59:03.794432247 +0000 UTC m=+450.236634551"
Nov 03 11:59:03.897820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:03.897758    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb" podStartSLOduration=1.368454842 podStartE2EDuration="8.897740725s" totalImagesPullingTime="7.529285883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:03.896891715 +0000 UTC m=+450.339094030" watchObservedRunningTime="2026-11-03 11:59:03.897740725 +0000 UTC m=+450.339943032"
Nov 03 11:59:05.611949 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:05.611915    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:05.775258 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:05.775197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:07.610446 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:07.610411    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:07.759087 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:07.759052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:07.918576 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:07.918516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:08.069318 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:08.069272    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:12.681488 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:12.681409    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-q6gzj"
Nov 03 11:59:23.804410 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.804361    5784 generic.go:356] "Generic (PLEG): container finished" podID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerID="3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed" exitCode=0
Nov 03 11:59:23.804844 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.804438    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerDied","Data":"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"}
Nov 03 11:59:23.804844 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.804821    5784 scope.go:122] "RemoveContainer" containerID="3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"
Nov 03 11:59:23.805967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.805936    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:59:23.806028 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.806006    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:59:23.806062 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:23.806041    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 11:59:24.811853 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:24.811816    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerStarted","Data":"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"}
Nov 03 11:59:25.178743 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:25.178704    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:59:25.178947 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:25.178922    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 11:59:25.179013 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:25.178987    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 11:59:26.179425 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.179358    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 11:59:26.179983 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.179445    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 11:59:26.471935 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:59:26.471845    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 11:59:26.655856 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.655823    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:26.900895 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.900859    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:26.941649 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.941615    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:26.945916 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.945891    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:26.967157 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.967119    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pj87\" (UniqueName: \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\") pod \"authorino-6f6944bcdd-t4nmx\" (UID: \"dc283260-bb03-4162-804f-d1866eeb3335\") " pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:26.995272 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:26.995240    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:27.068505 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.068471    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pj87\" (UniqueName: \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\") pod \"authorino-6f6944bcdd-t4nmx\" (UID: \"dc283260-bb03-4162-804f-d1866eeb3335\") " pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:27.134592 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.134560    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pj87\" (UniqueName: \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\") pod \"authorino-6f6944bcdd-t4nmx\" (UID: \"dc283260-bb03-4162-804f-d1866eeb3335\") " pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:27.178936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.178846    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 11:59:27.178936 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.178898    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 11:59:27.258193 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.258146    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:27.307137 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.307099    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:27.348547 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.346990    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:27.353720 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.353016    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:27.371855 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.371824    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hs8\" (UniqueName: \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\") pod \"authorino-695774bb5-nh9cb\" (UID: \"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a\") " pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:27.398725 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.398678    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:27.472998 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.472899    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hs8\" (UniqueName: \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\") pod \"authorino-695774bb5-nh9cb\" (UID: \"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a\") " pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:27.543535 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.543485    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hs8\" (UniqueName: \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\") pod \"authorino-695774bb5-nh9cb\" (UID: \"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a\") " pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:27.642065 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.641865    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:27.645236 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:59:27.645202    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc283260_bb03_4162_804f_d1866eeb3335.slice/crio-3ee498da5052d8586add497aecf6a897e7036e199a21dc20e6d16f23b7195339 WatchSource:0}: Error finding container 3ee498da5052d8586add497aecf6a897e7036e199a21dc20e6d16f23b7195339: Status 404 returned error can't find the container with id 3ee498da5052d8586add497aecf6a897e7036e199a21dc20e6d16f23b7195339
Nov 03 11:59:27.668633 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.668600    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:27.742644 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.742526    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:27.766300 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.766266    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 11:59:27.772704 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.772673    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:27.842904 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.842868    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 11:59:27.846990 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.846953    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 11:59:27.859898 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.859863    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" event={"ID":"dc283260-bb03-4162-804f-d1866eeb3335","Type":"ContainerStarted","Data":"3ee498da5052d8586add497aecf6a897e7036e199a21dc20e6d16f23b7195339"}
Nov 03 11:59:27.877063 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.877027    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:27.877218 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.877084    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87t85\" (UniqueName: \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:27.978581 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.978538    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87t85\" (UniqueName: \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:27.978760 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.978628    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:27.981111 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:27.981080    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:28.042733 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.042652    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87t85\" (UniqueName: \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\") pod \"authorino-c96647447-h2rjg\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") " pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:28.058950 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.058926    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:28.060815 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:59:28.060785    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94e7b123_cd4a_4ce6_9d72_b4b399f0d63a.slice/crio-98bfe2ff222507e79956521b2a7e9bafa113de2478222c3e2788b96327de7374 WatchSource:0}: Error finding container 98bfe2ff222507e79956521b2a7e9bafa113de2478222c3e2788b96327de7374: Status 404 returned error can't find the container with id 98bfe2ff222507e79956521b2a7e9bafa113de2478222c3e2788b96327de7374
Nov 03 11:59:28.089988 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.089955    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 11:59:28.178680 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.178642    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 11:59:28.178852 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.178707    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 11:59:28.483405 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.482587    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 11:59:28.483859 ip-10-0-139-140 kubenswrapper[5784]: W1103 11:59:28.483822    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa80b107_3f08_482c_81f4_c8f8d790a834.slice/crio-bbe8e836f0a681e797aa5be97207ac62b24a2486212c0d4ee9f42fe06105b8f2 WatchSource:0}: Error finding container bbe8e836f0a681e797aa5be97207ac62b24a2486212c0d4ee9f42fe06105b8f2: Status 404 returned error can't find the container with id bbe8e836f0a681e797aa5be97207ac62b24a2486212c0d4ee9f42fe06105b8f2
Nov 03 11:59:28.867658 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.867579    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695774bb5-nh9cb" event={"ID":"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a","Type":"ContainerStarted","Data":"98bfe2ff222507e79956521b2a7e9bafa113de2478222c3e2788b96327de7374"}
Nov 03 11:59:28.869115 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:28.869088    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c96647447-h2rjg" event={"ID":"fa80b107-3f08-482c-81f4-c8f8d790a834","Type":"ContainerStarted","Data":"bbe8e836f0a681e797aa5be97207ac62b24a2486212c0d4ee9f42fe06105b8f2"}
Nov 03 11:59:29.178599 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:29.178549    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Nov 03 11:59:29.178806 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:29.178617    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Nov 03 11:59:30.183147 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.183116    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:59:30.183621 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.183419    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:59:30.184318 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.184301    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 11:59:30.883337 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.883290    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695774bb5-nh9cb" event={"ID":"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a","Type":"ContainerStarted","Data":"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"}
Nov 03 11:59:30.883337 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.883320    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-695774bb5-nh9cb" podUID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" containerName="authorino" containerID="cri-o://86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9" gracePeriod=30
Nov 03 11:59:30.885081 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.885050    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c96647447-h2rjg" event={"ID":"fa80b107-3f08-482c-81f4-c8f8d790a834","Type":"ContainerStarted","Data":"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"}
Nov 03 11:59:30.887535 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.887503    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" podUID="dc283260-bb03-4162-804f-d1866eeb3335" containerName="authorino" containerID="cri-o://85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4" gracePeriod=30
Nov 03 11:59:30.887648 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.887623    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" event={"ID":"dc283260-bb03-4162-804f-d1866eeb3335","Type":"ContainerStarted","Data":"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"}
Nov 03 11:59:30.991912 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:30.991844    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-695774bb5-nh9cb" podStartSLOduration=1.765412143 podStartE2EDuration="3.991822686s" totalImagesPullingTime="2.226410543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:30.988157376 +0000 UTC m=+477.430359684" watchObservedRunningTime="2026-11-03 11:59:30.991822686 +0000 UTC m=+477.434024995"
Nov 03 11:59:31.102607 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.102064    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c96647447-h2rjg" podStartSLOduration=2.240554363 podStartE2EDuration="4.102044101s" totalImagesPullingTime="1.861489738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:31.099092423 +0000 UTC m=+477.541294755" watchObservedRunningTime="2026-11-03 11:59:31.102044101 +0000 UTC m=+477.544246407"
Nov 03 11:59:31.119704 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.119669    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:59:31.119908 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.119885    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" podUID="8c72d989-d9d6-4780-99fd-4388997af0e6" containerName="authorino" containerID="cri-o://93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe" gracePeriod=30
Nov 03 11:59:31.207163 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.207107    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" podStartSLOduration=2.593045331 podStartE2EDuration="5.207088378s" totalImagesPullingTime="2.614043047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 11:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 11:59:31.206293411 +0000 UTC m=+477.648495718" watchObservedRunningTime="2026-11-03 11:59:31.207088378 +0000 UTC m=+477.649290684"
Nov 03 11:59:31.249780 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.249753    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:31.263279 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.263245    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:31.320540 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.313296    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\" (UniqueName: \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\") pod \"dc283260-bb03-4162-804f-d1866eeb3335\" (UID: \"dc283260-bb03-4162-804f-d1866eeb3335\") "
Nov 03 11:59:31.320540 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.313422    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\" (UniqueName: \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\") pod \"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a\" (UID: \"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a\") "
Nov 03 11:59:31.321144 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.321105    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87" pod "dc283260-bb03-4162-804f-d1866eeb3335" (UID: "dc283260-bb03-4162-804f-d1866eeb3335"). InnerVolumeSpecName "kube-api-access-9pj87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:59:31.323127 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.323090    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8" pod "94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" (UID: "94e7b123-cd4a-4ce6-9d72-b4b399f0d63a"). InnerVolumeSpecName "kube-api-access-g5hs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:59:31.416821 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.416786    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hs8\" (UniqueName: \"kubernetes.io/projected/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a-kube-api-access-g5hs8\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:31.416821 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.416822    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pj87\" (UniqueName: \"kubernetes.io/projected/dc283260-bb03-4162-804f-d1866eeb3335-kube-api-access-9pj87\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:31.423820 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.423785    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:59:31.517675 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.517640    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\" (UniqueName: \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\") pod \"8c72d989-d9d6-4780-99fd-4388997af0e6\" (UID: \"8c72d989-d9d6-4780-99fd-4388997af0e6\") "
Nov 03 11:59:31.519979 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.519945    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh" pod "8c72d989-d9d6-4780-99fd-4388997af0e6" (UID: "8c72d989-d9d6-4780-99fd-4388997af0e6"). InnerVolumeSpecName "kube-api-access-d89rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 11:59:31.619022 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.618985    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d89rh\" (UniqueName: \"kubernetes.io/projected/8c72d989-d9d6-4780-99fd-4388997af0e6-kube-api-access-d89rh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 11:59:31.896786 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.896748    5784 generic.go:356] "Generic (PLEG): container finished" podID="dc283260-bb03-4162-804f-d1866eeb3335" containerID="85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4" exitCode=0
Nov 03 11:59:31.896786 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.896788    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx"
Nov 03 11:59:31.897005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.896833    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" event={"ID":"dc283260-bb03-4162-804f-d1866eeb3335","Type":"ContainerDied","Data":"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"}
Nov 03 11:59:31.897005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.896871    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-t4nmx" event={"ID":"dc283260-bb03-4162-804f-d1866eeb3335","Type":"ContainerDied","Data":"3ee498da5052d8586add497aecf6a897e7036e199a21dc20e6d16f23b7195339"}
Nov 03 11:59:31.897005 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.896893    5784 scope.go:122] "RemoveContainer" containerID="85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"
Nov 03 11:59:31.898283 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.898260    5784 generic.go:356] "Generic (PLEG): container finished" podID="8c72d989-d9d6-4780-99fd-4388997af0e6" containerID="93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe" exitCode=0
Nov 03 11:59:31.898414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.898306    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f"
Nov 03 11:59:31.898414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.898344    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" event={"ID":"8c72d989-d9d6-4780-99fd-4388997af0e6","Type":"ContainerDied","Data":"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"}
Nov 03 11:59:31.898414 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.898406    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-xz86f" event={"ID":"8c72d989-d9d6-4780-99fd-4388997af0e6","Type":"ContainerDied","Data":"405ccfde6b4e0c456dbe5e7a395db4ec030acf2b5a95e71e3eb7e3945c7351b9"}
Nov 03 11:59:31.900120 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.900022    5784 generic.go:356] "Generic (PLEG): container finished" podID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" containerID="86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9" exitCode=0
Nov 03 11:59:31.900120 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.900048    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695774bb5-nh9cb" event={"ID":"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a","Type":"ContainerDied","Data":"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"}
Nov 03 11:59:31.900120 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.900065    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-695774bb5-nh9cb"
Nov 03 11:59:31.900120 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.900077    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-695774bb5-nh9cb" event={"ID":"94e7b123-cd4a-4ce6-9d72-b4b399f0d63a","Type":"ContainerDied","Data":"98bfe2ff222507e79956521b2a7e9bafa113de2478222c3e2788b96327de7374"}
Nov 03 11:59:31.907128 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.907108    5784 scope.go:122] "RemoveContainer" containerID="85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"
Nov 03 11:59:31.907387 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:59:31.907358    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4\": container with ID starting with 85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4 not found: ID does not exist" containerID="85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"
Nov 03 11:59:31.907439 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.907401    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4"} err="failed to get container status \"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4\": rpc error: code = NotFound desc = could not find container \"85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4\": container with ID starting with 85db6790b4d86f4bb56b74fada07ff32a8fe46069439ac6e326b38aca9b8b8a4 not found: ID does not exist"
Nov 03 11:59:31.907439 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.907419    5784 scope.go:122] "RemoveContainer" containerID="93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"
Nov 03 11:59:31.915704 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.915686    5784 scope.go:122] "RemoveContainer" containerID="93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"
Nov 03 11:59:31.915960 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:59:31.915942    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe\": container with ID starting with 93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe not found: ID does not exist" containerID="93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"
Nov 03 11:59:31.916006 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.915971    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe"} err="failed to get container status \"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe\": rpc error: code = NotFound desc = could not find container \"93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe\": container with ID starting with 93237e4e69c5d9f1de1448e9494dafe6bbd6439eb218fc5a885be2bb2fb79efe not found: ID does not exist"
Nov 03 11:59:31.916006 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.915993    5784 scope.go:122] "RemoveContainer" containerID="86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"
Nov 03 11:59:31.924026 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.924011    5784 scope.go:122] "RemoveContainer" containerID="86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"
Nov 03 11:59:31.924299 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:59:31.924279    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9\": container with ID starting with 86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9 not found: ID does not exist" containerID="86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"
Nov 03 11:59:31.924354 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:31.924310    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9"} err="failed to get container status \"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9\": rpc error: code = NotFound desc = could not find container \"86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9\": container with ID starting with 86c9787331a05fa2cedbfde6375e37282b3b6252e61f9b0e9d8390ad2f9d6ea9 not found: ID does not exist"
Nov 03 11:59:32.048778 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.048688    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:59:32.052352 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.052325    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-xz86f"]
Nov 03 11:59:32.152614 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.152574    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:32.156794 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.156761    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-t4nmx"]
Nov 03 11:59:32.256967 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.256928    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:32.262192 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:32.262165    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-695774bb5-nh9cb"]
Nov 03 11:59:33.687934 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:33.687902    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c72d989-d9d6-4780-99fd-4388997af0e6" path="/var/lib/kubelet/pods/8c72d989-d9d6-4780-99fd-4388997af0e6/volumes"
Nov 03 11:59:33.688304 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:33.688234    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" path="/var/lib/kubelet/pods/94e7b123-cd4a-4ce6-9d72-b4b399f0d63a/volumes"
Nov 03 11:59:33.688572 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:33.688559    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc283260-bb03-4162-804f-d1866eeb3335" path="/var/lib/kubelet/pods/dc283260-bb03-4162-804f-d1866eeb3335/volumes"
Nov 03 11:59:34.696811 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:34.696781    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 11:59:34.700277 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:34.700250    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-np6xb"
Nov 03 11:59:43.346508 ip-10-0-139-140 kubenswrapper[5784]: I1103 11:59:43.346471    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 11:59:56.471171 ip-10-0-139-140 kubenswrapper[5784]: E1103 11:59:56.471132    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:00:12.437870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.437834    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"]
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438270    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8c72d989-d9d6-4780-99fd-4388997af0e6" containerName="authorino"
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438281    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c72d989-d9d6-4780-99fd-4388997af0e6" containerName="authorino"
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438293    5784 cpu_manager.go:414] "containerMap: removing container" podUID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" containerName="authorino"
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438298    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" containerName="authorino"
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438316    5784 cpu_manager.go:414] "containerMap: removing container" podUID="dc283260-bb03-4162-804f-d1866eeb3335" containerName="authorino"
Nov 03 12:00:12.438363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438320    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc283260-bb03-4162-804f-d1866eeb3335" containerName="authorino"
Nov 03 12:00:12.438624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438401    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc283260-bb03-4162-804f-d1866eeb3335" containerName="authorino"
Nov 03 12:00:12.438624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438415    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c72d989-d9d6-4780-99fd-4388997af0e6" containerName="authorino"
Nov 03 12:00:12.438624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.438421    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e7b123-cd4a-4ce6-9d72-b4b399f0d63a" containerName="authorino"
Nov 03 12:00:12.441138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.441120    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.495855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.495822    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"]
Nov 03 12:00:12.521680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.521614    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76lfg\" (UniqueName: \"kubernetes.io/projected/4e96077b-bec2-45f0-97bc-58f8f620f925-kube-api-access-76lfg\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.521841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.521724    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96077b-bec2-45f0-97bc-58f8f620f925-cert\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.622479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.622442    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76lfg\" (UniqueName: \"kubernetes.io/projected/4e96077b-bec2-45f0-97bc-58f8f620f925-kube-api-access-76lfg\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.622667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.622520    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96077b-bec2-45f0-97bc-58f8f620f925-cert\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.625064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.625040    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e96077b-bec2-45f0-97bc-58f8f620f925-cert\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.676299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.676259    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76lfg\" (UniqueName: \"kubernetes.io/projected/4e96077b-bec2-45f0-97bc-58f8f620f925-kube-api-access-76lfg\") pod \"llmisvc-controller-manager-6466c554d8-rxmxg\" (UID: \"4e96077b-bec2-45f0-97bc-58f8f620f925\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:12.754430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:12.754309    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:13.119047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:13.119019    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"]
Nov 03 12:00:13.120980 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:00:13.120951    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e96077b_bec2_45f0_97bc_58f8f620f925.slice/crio-5132f3090a0c767a60d5c3e1d86d5ff9fa8d78e57cd9dae7e1f7037406e733c0 WatchSource:0}: Error finding container 5132f3090a0c767a60d5c3e1d86d5ff9fa8d78e57cd9dae7e1f7037406e733c0: Status 404 returned error can't find the container with id 5132f3090a0c767a60d5c3e1d86d5ff9fa8d78e57cd9dae7e1f7037406e733c0
Nov 03 12:00:13.146473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:13.146426    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg" event={"ID":"4e96077b-bec2-45f0-97bc-58f8f620f925","Type":"ContainerStarted","Data":"5132f3090a0c767a60d5c3e1d86d5ff9fa8d78e57cd9dae7e1f7037406e733c0"}
Nov 03 12:00:14.154675 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:14.154633    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg" event={"ID":"4e96077b-bec2-45f0-97bc-58f8f620f925","Type":"ContainerStarted","Data":"4bbfbd2ab6b5b8a84ecad5677f2f829e070d4e1a4bde0da94f328cd4633fa4ef"}
Nov 03 12:00:14.155083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:14.154708    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:14.261842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:14.261168    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg" podStartSLOduration=1.557785055 podStartE2EDuration="2.261148626s" totalImagesPullingTime="703.363571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:00:14.260251786 +0000 UTC m=+520.702454091" watchObservedRunningTime="2026-11-03 12:00:14.261148626 +0000 UTC m=+520.703350934"
Nov 03 12:00:22.479721 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:22.479643    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:26.470904 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:00:26.470867    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:00:43.672199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:43.672110    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:45.162710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.162677    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-rxmxg"
Nov 03 12:00:45.319219 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.319179    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 12:00:45.319477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.319448    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" podUID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" containerName="manager" containerID="cri-o://c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d" gracePeriod=30
Nov 03 12:00:45.563663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.563639    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 12:00:45.745004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.744915    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\" (UniqueName: \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\") pod \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") "
Nov 03 12:00:45.745004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.744962    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\" (UniqueName: \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\") pod \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\" (UID: \"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7\") "
Nov 03 12:00:45.747209 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.747174    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert" pod "801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" (UID: "801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:00:45.747337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.747308    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l" pod "801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" (UID: "801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7"). InnerVolumeSpecName "kube-api-access-ftt2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:00:45.846605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.846563    5784 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:00:45.852984 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:45.846600    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftt2l\" (UniqueName: \"kubernetes.io/projected/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7-kube-api-access-ftt2l\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:00:46.344710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.344667    5784 generic.go:356] "Generic (PLEG): container finished" podID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" containerID="c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d" exitCode=0
Nov 03 12:00:46.345139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.344738    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"
Nov 03 12:00:46.345139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.344762    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" event={"ID":"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7","Type":"ContainerDied","Data":"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"}
Nov 03 12:00:46.345139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.344837    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb" event={"ID":"801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7","Type":"ContainerDied","Data":"e89ab693080e85cab9da4f640d542adad8fd9433a71659668b3262aaf469ff11"}
Nov 03 12:00:46.345139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.344864    5784 scope.go:122] "RemoveContainer" containerID="c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"
Nov 03 12:00:46.357111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.357083    5784 scope.go:122] "RemoveContainer" containerID="c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"
Nov 03 12:00:46.357480 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:00:46.357456    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d\": container with ID starting with c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d not found: ID does not exist" containerID="c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"
Nov 03 12:00:46.357564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.357490    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d"} err="failed to get container status \"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d\": rpc error: code = NotFound desc = could not find container \"c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d\": container with ID starting with c5e45f823ebbc554d9098f010a4584ecfd8ae44b37c2c4f38b100d4269af4e8d not found: ID does not exist"
Nov 03 12:00:46.496395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.496335    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 12:00:46.542499 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.542459    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-bpnkb"]
Nov 03 12:00:46.670616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.670579    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:46.824663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:46.824626    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:47.680729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:47.680686    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" path="/var/lib/kubelet/pods/801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7/volumes"
Nov 03 12:00:55.183275 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:55.183231    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:55.352962 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:00:55.352931    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:00:56.470929 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:00:56.470890    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:01:03.446628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:03.446587    5784 generic.go:356] "Generic (PLEG): container finished" podID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerID="e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31" exitCode=1
Nov 03 12:01:03.447011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:03.446660    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerDied","Data":"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"}
Nov 03 12:01:03.447053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:03.447035    5784 scope.go:122] "RemoveContainer" containerID="e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"
Nov 03 12:01:04.454583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:04.454545    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerStarted","Data":"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"}
Nov 03 12:01:04.455012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:04.454790    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 12:01:15.462996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:15.462963    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 12:01:18.710472 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:18.710435    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:19.137053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:19.137014    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:20.355479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:20.355443    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:20.527941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:20.527908    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:25.729569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:25.729536    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:26.472452 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:01:26.472410    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:01:28.293043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:28.293010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:33.760725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:33.760683    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:01:33.761174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:33.760864    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:01:33.960253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:33.960219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:37.252186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:37.252130    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:40.368855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:40.368815    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:43.989895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:43.989856    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:44.184479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:44.184439    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:47.776648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:47.776612    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:50.757359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:50.757311    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:54.092510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:01:54.092470    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:01:56.470831 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:01:56.470796    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:02:20.784501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.784415    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fccf55576-2gn95"]
Nov 03 12:02:20.784988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.784894    5784 cpu_manager.go:414] "containerMap: removing container" podUID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" containerName="manager"
Nov 03 12:02:20.784988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.784910    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" containerName="manager"
Nov 03 12:02:20.785061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.784998    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="801bf2d9-58ac-4a4f-8876-c1f68dd0b5b7" containerName="manager"
Nov 03 12:02:20.787200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.787183    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:20.840839 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.840803    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fccf55576-2gn95"]
Nov 03 12:02:20.883489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.883445    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9117e63-a8c2-4c65-a7d0-adead6725e93-tls-cert\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:20.883670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.883500    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnsdz\" (UniqueName: \"kubernetes.io/projected/e9117e63-a8c2-4c65-a7d0-adead6725e93-kube-api-access-bnsdz\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:20.984482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.984447    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9117e63-a8c2-4c65-a7d0-adead6725e93-tls-cert\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:20.984482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.984495    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsdz\" (UniqueName: \"kubernetes.io/projected/e9117e63-a8c2-4c65-a7d0-adead6725e93-kube-api-access-bnsdz\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:20.986934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:20.986913    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e9117e63-a8c2-4c65-a7d0-adead6725e93-tls-cert\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:21.039866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.039792    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsdz\" (UniqueName: \"kubernetes.io/projected/e9117e63-a8c2-4c65-a7d0-adead6725e93-kube-api-access-bnsdz\") pod \"authorino-5fccf55576-2gn95\" (UID: \"e9117e63-a8c2-4c65-a7d0-adead6725e93\") " pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:21.098129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.098093    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fccf55576-2gn95"
Nov 03 12:02:21.466892 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.466858    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fccf55576-2gn95"]
Nov 03 12:02:21.469325 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:02:21.469297    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9117e63_a8c2_4c65_a7d0_adead6725e93.slice/crio-946fbb9a83b6243f09759c6a0e40f6ec83d2d4f10776fd65445010af1c7dd486 WatchSource:0}: Error finding container 946fbb9a83b6243f09759c6a0e40f6ec83d2d4f10776fd65445010af1c7dd486: Status 404 returned error can't find the container with id 946fbb9a83b6243f09759c6a0e40f6ec83d2d4f10776fd65445010af1c7dd486
Nov 03 12:02:21.504252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.504218    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:21.664786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.664756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:21.948687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:21.948644    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fccf55576-2gn95" event={"ID":"e9117e63-a8c2-4c65-a7d0-adead6725e93","Type":"ContainerStarted","Data":"946fbb9a83b6243f09759c6a0e40f6ec83d2d4f10776fd65445010af1c7dd486"}
Nov 03 12:02:23.130251 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:23.130219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:23.292238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:23.292202    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:25.973547 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:25.973506    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fccf55576-2gn95" event={"ID":"e9117e63-a8c2-4c65-a7d0-adead6725e93","Type":"ContainerStarted","Data":"860bf9bad337ddcad874c61c73faecafbcaf83ed3eb91ebf8708a18926588782"}
Nov 03 12:02:26.082215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.082142    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fccf55576-2gn95" podStartSLOduration=1.968779122 podStartE2EDuration="6.082125434s" totalImagesPullingTime="4.113346312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:02:26.079605013 +0000 UTC m=+652.521807338" watchObservedRunningTime="2026-11-03 12:02:26.082125434 +0000 UTC m=+652.524327751"
Nov 03 12:02:26.103807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.103775    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 12:02:26.103989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.103968    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c96647447-h2rjg" podUID="fa80b107-3f08-482c-81f4-c8f8d790a834" containerName="authorino" containerID="cri-o://56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b" gracePeriod=30
Nov 03 12:02:26.367306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.367267    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 12:02:26.438909 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.438869    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\" (UniqueName: \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\") pod \"fa80b107-3f08-482c-81f4-c8f8d790a834\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") "
Nov 03 12:02:26.439102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.438957    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\" (UniqueName: \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\") pod \"fa80b107-3f08-482c-81f4-c8f8d790a834\" (UID: \"fa80b107-3f08-482c-81f4-c8f8d790a834\") "
Nov 03 12:02:26.441065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.441033    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85" pod "fa80b107-3f08-482c-81f4-c8f8d790a834" (UID: "fa80b107-3f08-482c-81f4-c8f8d790a834"). InnerVolumeSpecName "kube-api-access-87t85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:02:26.450502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.450457    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert" pod "fa80b107-3f08-482c-81f4-c8f8d790a834" (UID: "fa80b107-3f08-482c-81f4-c8f8d790a834"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:02:26.479588 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:02:26.479498    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:02:26.540756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.540714    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87t85\" (UniqueName: \"kubernetes.io/projected/fa80b107-3f08-482c-81f4-c8f8d790a834-kube-api-access-87t85\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:02:26.540756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.540749    5784 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fa80b107-3f08-482c-81f4-c8f8d790a834-tls-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:02:26.997402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.996656    5784 generic.go:356] "Generic (PLEG): container finished" podID="fa80b107-3f08-482c-81f4-c8f8d790a834" containerID="56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b" exitCode=0
Nov 03 12:02:26.997900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.997439    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c96647447-h2rjg"
Nov 03 12:02:26.998946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.998916    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c96647447-h2rjg" event={"ID":"fa80b107-3f08-482c-81f4-c8f8d790a834","Type":"ContainerDied","Data":"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"}
Nov 03 12:02:26.999242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.999182    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c96647447-h2rjg" event={"ID":"fa80b107-3f08-482c-81f4-c8f8d790a834","Type":"ContainerDied","Data":"bbe8e836f0a681e797aa5be97207ac62b24a2486212c0d4ee9f42fe06105b8f2"}
Nov 03 12:02:26.999408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:26.999226    5784 scope.go:122] "RemoveContainer" containerID="56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"
Nov 03 12:02:27.013116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:27.013014    5784 scope.go:122] "RemoveContainer" containerID="56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"
Nov 03 12:02:27.013348 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:02:27.013329    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b\": container with ID starting with 56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b not found: ID does not exist" containerID="56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"
Nov 03 12:02:27.013432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:27.013363    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b"} err="failed to get container status \"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b\": rpc error: code = NotFound desc = could not find container \"56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b\": container with ID starting with 56eea9dce1c9fce06f3e39f5fbf10bc2d07814746b72efc224c40a8a5abce60b not found: ID does not exist"
Nov 03 12:02:27.151225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:27.151188    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 12:02:27.153466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:27.153437    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c96647447-h2rjg"]
Nov 03 12:02:27.681833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:27.681796    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa80b107-3f08-482c-81f4-c8f8d790a834" path="/var/lib/kubelet/pods/fa80b107-3f08-482c-81f4-c8f8d790a834/volumes"
Nov 03 12:02:28.107215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:28.107011    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:44.301602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:02:44.301561    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:02:56.471036 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:02:56.470993    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:03:05.930822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:05.930792    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:06.107980 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:06.107950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:06.278292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:06.278191    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:06.485655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:06.485606    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:06.708494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:06.708464    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:06.969443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:06.964600    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:07.124957 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:07.124922    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:07.290858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:07.290777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:07.618899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:07.618815    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:07.804261 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:07.804236    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:07.967504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:07.967472    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:08.982166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:08.982128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:09.161045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:09.161015    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:10.296623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:10.296593    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:10.475324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:10.475291    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:10.627186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:10.627157    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:10.847770 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:10.847737    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:11.065900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:11.065801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:11.288763 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:11.288729    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:11.852320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:11.852284    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:12.049299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:12.049266    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:12.755843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:12.755811    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:12.914659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:12.914631    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:13.086069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:13.085983    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:13.716202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:13.716170    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:13.904601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:13.904564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:14.127942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:14.127878    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:14.282332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:14.282283    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:16.025937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:16.025907    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:16.219032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:16.218992    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:16.390507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:16.390475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:16.547692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:16.547657    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:18.632856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:18.632815    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:18.784309 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:18.784273    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:19.176776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.176740    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:03:19.177269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.177250    5784 cpu_manager.go:414] "containerMap: removing container" podUID="fa80b107-3f08-482c-81f4-c8f8d790a834" containerName="authorino"
Nov 03 12:03:19.177353 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.177271    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa80b107-3f08-482c-81f4-c8f8d790a834" containerName="authorino"
Nov 03 12:03:19.177478 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.177422    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa80b107-3f08-482c-81f4-c8f8d790a834" containerName="authorino"
Nov 03 12:03:19.180876 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.180854    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.233642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.233613    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:03:19.259020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.258988    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.259198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.259025    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9ld\" (UniqueName: \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.276640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.276599    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1319e7ef-dockercfg-7jzck\""
Nov 03 12:03:19.276640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.276607    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1319e7ef\""
Nov 03 12:03:19.360697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.360649    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.360894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.360704    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9ld\" (UniqueName: \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.361479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.361459    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.416824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.416789    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9ld\" (UniqueName: \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\") pod \"payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") " pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.494560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.494470    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:19.825079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.824999    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:19.858838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.858813    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:03:19.860417 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:03:19.860373    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbd6c92_5d8a_47b3_9f5a_a13fd23ec7f7.slice/crio-2fc780261769290de1f54f510ab96e29ec585fea119fc2ca7cfdd064c2b57fe9 WatchSource:0}: Error finding container 2fc780261769290de1f54f510ab96e29ec585fea119fc2ca7cfdd064c2b57fe9: Status 404 returned error can't find the container with id 2fc780261769290de1f54f510ab96e29ec585fea119fc2ca7cfdd064c2b57fe9
Nov 03 12:03:19.862303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:19.862289    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:03:20.025461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:20.025424    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:20.193625 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:20.193590    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:20.308161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:20.308120    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" event={"ID":"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7","Type":"ContainerStarted","Data":"2fc780261769290de1f54f510ab96e29ec585fea119fc2ca7cfdd064c2b57fe9"}
Nov 03 12:03:20.905904 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:20.905869    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:21.069267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.069232    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:21.222042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.221961    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:21.463060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.463024    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:03:21.466843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.466815    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.520457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.516831    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f732077f\""
Nov 03 12:03:21.524117 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.524084    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:03:21.562479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.562437    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f732077f-dockercfg-9rflh\""
Nov 03 12:03:21.584806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.584768    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9xf\" (UniqueName: \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.584964 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.584890    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.685783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.685753    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.685977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.685874    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9xf\" (UniqueName: \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.686716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.686691    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.738647 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.738582    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9xf\" (UniqueName: \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\") pod \"payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") " pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:21.782723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:21.782691    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:22.159609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:22.159580    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:03:22.161032 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:03:22.160997    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fb29bb_4a43_4af4_b26a_f250e3d59be1.slice/crio-fa7ee4be9ab2e7494211330546636fbd2e3795fc0008eebe87a382fcbcdf045f WatchSource:0}: Error finding container fa7ee4be9ab2e7494211330546636fbd2e3795fc0008eebe87a382fcbcdf045f: Status 404 returned error can't find the container with id fa7ee4be9ab2e7494211330546636fbd2e3795fc0008eebe87a382fcbcdf045f
Nov 03 12:03:22.324210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:22.324175    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" event={"ID":"66fb29bb-4a43-4af4-b26a-f250e3d59be1","Type":"ContainerStarted","Data":"fa7ee4be9ab2e7494211330546636fbd2e3795fc0008eebe87a382fcbcdf045f"}
Nov 03 12:03:23.244398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.244351    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:23.334489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.334361    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" event={"ID":"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7","Type":"ContainerStarted","Data":"78cab4cae5c0b240aa76b7268bba39e8e5a51a6e0f989d4ec116e14d3db1de2e"}
Nov 03 12:03:23.334695 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.334644    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:23.336081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.336056    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" event={"ID":"66fb29bb-4a43-4af4-b26a-f250e3d59be1","Type":"ContainerStarted","Data":"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"}
Nov 03 12:03:23.336211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.336170    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:23.398702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.398669    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:23.441870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.441806    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" podStartSLOduration=1.897751796 podStartE2EDuration="4.441791333s" totalImagesPullingTime="2.544039537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:23.440051972 +0000 UTC m=+709.882254295" watchObservedRunningTime="2026-11-03 12:03:23.441791333 +0000 UTC m=+709.883993639"
Nov 03 12:03:23.555961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:23.555911    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" podStartSLOduration=1.663905626 podStartE2EDuration="2.555893896s" totalImagesPullingTime="891.98827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:23.55324945 +0000 UTC m=+709.995451755" watchObservedRunningTime="2026-11-03 12:03:23.555893896 +0000 UTC m=+709.998096201"
Nov 03 12:03:25.661395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:25.661347    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:25.826649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:25.826600    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:25.985693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:25.985617    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:26.476009 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:03:26.475968    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:03:30.012216 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:30.012168    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:31.969781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:31.969739    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:32.150347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:32.150255    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:34.345149 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:34.345108    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:03:34.345674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:34.345447    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:03:34.480390 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:34.480345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:34.641715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:34.641677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:34.791828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:34.791802    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:38.768043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.768012    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:03:38.772557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.772537    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.862842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.862809    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.863012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.862986    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.863054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.863038    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.863092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.863070    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pzh\" (UniqueName: \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.868844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.868821    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-9ab28c7a-serving-cert\""
Nov 03 12:03:38.868981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.868951    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vv49\""
Nov 03 12:03:38.868981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.868951    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 12:03:38.871438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.871417    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:03:38.964162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.964122    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.964337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.964210    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.964337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.964235    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.964337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.964253    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pzh\" (UniqueName: \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.964491 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:03:38.964393    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-serving-cert: secret "maas-api-e2e-derive-9ab28c7a-serving-cert" not found
Nov 03 12:03:38.964491 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:03:38.964474    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls podName:6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa nodeName:}" failed. No retries permitted until 2026-11-03 12:03:39.46445037 +0000 UTC m=+725.906652655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls") pod "maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" (UID: "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa") : secret "maas-api-e2e-derive-9ab28c7a-serving-cert" not found
Nov 03 12:03:38.964928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.964910    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.966532 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.966514    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:38.970318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:38.970290    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:39.020166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.020093    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pzh\" (UniqueName: \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:39.134019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.133961    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:39.291643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.291558    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:39.470753 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.470718    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:39.473127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.473104    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") pod \"maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:39.685996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:39.685966    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:40.051847 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:40.051818    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:03:40.053488 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:03:40.053463    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e094aa8_fbf9_4c6e_9fb0_142bcb0d3daa.slice/crio-fadbaf6fc584b81efa22c769839bae41907b2a9d240ba31dd53bde799f4c3cf1 WatchSource:0}: Error finding container fadbaf6fc584b81efa22c769839bae41907b2a9d240ba31dd53bde799f4c3cf1: Status 404 returned error can't find the container with id fadbaf6fc584b81efa22c769839bae41907b2a9d240ba31dd53bde799f4c3cf1
Nov 03 12:03:40.438611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:40.438578    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" event={"ID":"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa","Type":"ContainerStarted","Data":"fadbaf6fc584b81efa22c769839bae41907b2a9d240ba31dd53bde799f4c3cf1"}
Nov 03 12:03:43.469700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:43.469669    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" event={"ID":"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa","Type":"ContainerStarted","Data":"a2a5b986aa21d8ee9e3a238ae381eb329bd575cba1e2312b14db8f05d17a75dd"}
Nov 03 12:03:43.470081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:43.469807    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:43.581075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:43.581023    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" podStartSLOduration=2.279804477 podStartE2EDuration="5.581007906s" totalImagesPullingTime="3.301203429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:03:43.579785579 +0000 UTC m=+730.021987908" watchObservedRunningTime="2026-11-03 12:03:43.581007906 +0000 UTC m=+730.023210211"
Nov 03 12:03:43.636975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:43.636911    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:44.632116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:44.632082    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:48.020343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:48.020310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:49.490515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:49.490482    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:03:53.099980 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:53.099940    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:03:56.471471 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:03:56.471427    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:03:57.523583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:03:57.523550    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:02.293413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.293359    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"]
Nov 03 12:04:02.300719 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.300691    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.350440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.350405    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"]
Nov 03 12:04:02.393580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.393547    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 12:04:02.393749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.393617    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:04:02.397050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.397025    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.397147 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.397087    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh5z8\" (UniqueName: \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.397185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.397171    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.459598 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.459564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:02.498365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.498326    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.498585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.498464    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh5z8\" (UniqueName: \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.498585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.498520    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.499138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.499117    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.500896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.500875    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.551961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.551882    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh5z8\" (UniqueName: \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\") pod \"maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.613948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.613912    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:02.978071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:02.978041    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"]
Nov 03 12:04:02.979955 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:04:02.979925    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4b1928_46b0_415e_9ed3_3579fcb2b559.slice/crio-f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b WatchSource:0}: Error finding container f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b: Status 404 returned error can't find the container with id f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b
Nov 03 12:04:03.245344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:03.245253    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:04:03.595345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:03.595265    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7" event={"ID":"cc4b1928-46b0-415e-9ed3-3579fcb2b559","Type":"ContainerStarted","Data":"f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b"}
Nov 03 12:04:03.615259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:03.615231    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:03.774838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:03.774797    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:07.296490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:07.296459    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:08.245907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.245861    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" podUID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" containerName="payload-processing" containerID="cri-o://b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9" gracePeriod=25
Nov 03 12:04:08.495186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.495159    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:04:08.628835 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.628797    5784 generic.go:356] "Generic (PLEG): container finished" podID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" containerID="b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9" exitCode=0
Nov 03 12:04:08.629014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.628840    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"
Nov 03 12:04:08.629014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.628877    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" event={"ID":"66fb29bb-4a43-4af4-b26a-f250e3d59be1","Type":"ContainerDied","Data":"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"}
Nov 03 12:04:08.629014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.628934    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h" event={"ID":"66fb29bb-4a43-4af4-b26a-f250e3d59be1","Type":"ContainerDied","Data":"fa7ee4be9ab2e7494211330546636fbd2e3795fc0008eebe87a382fcbcdf045f"}
Nov 03 12:04:08.629014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.628968    5784 scope.go:122] "RemoveContainer" containerID="b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"
Nov 03 12:04:08.630649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.630624    5784 generic.go:356] "Generic (PLEG): container finished" podID="cc4b1928-46b0-415e-9ed3-3579fcb2b559" containerID="dc09140b5b241e69c0083bbd865283f1719473f515689912d849dd6c05e670de" exitCode=0
Nov 03 12:04:08.630755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.630703    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7" event={"ID":"cc4b1928-46b0-415e-9ed3-3579fcb2b559","Type":"ContainerDied","Data":"dc09140b5b241e69c0083bbd865283f1719473f515689912d849dd6c05e670de"}
Nov 03 12:04:08.639153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.639134    5784 scope.go:122] "RemoveContainer" containerID="b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"
Nov 03 12:04:08.639469 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:08.639446    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9\": container with ID starting with b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9 not found: ID does not exist" containerID="b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"
Nov 03 12:04:08.639549 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.639478    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9"} err="failed to get container status \"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9\": rpc error: code = NotFound desc = could not find container \"b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9\": container with ID starting with b53baf775c66fb8d98face5f0e21c95362ae57bb17b72d8448320f7b614713a9 not found: ID does not exist"
Nov 03 12:04:08.657234 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.657212    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\") pod \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") "
Nov 03 12:04:08.657345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.657283    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\" (UniqueName: \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\") pod \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\" (UID: \"66fb29bb-4a43-4af4-b26a-f250e3d59be1\") "
Nov 03 12:04:08.657579 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.657559    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume" pod "66fb29bb-4a43-4af4-b26a-f250e3d59be1" (UID: "66fb29bb-4a43-4af4-b26a-f250e3d59be1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:08.659356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.659334    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf" pod "66fb29bb-4a43-4af4-b26a-f250e3d59be1" (UID: "66fb29bb-4a43-4af4-b26a-f250e3d59be1"). InnerVolumeSpecName "kube-api-access-ps9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:08.758968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.758893    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9xf\" (UniqueName: \"kubernetes.io/projected/66fb29bb-4a43-4af4-b26a-f250e3d59be1-kube-api-access-ps9xf\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:08.758968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:08.758920    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66fb29bb-4a43-4af4-b26a-f250e3d59be1-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:09.041398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.041297    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:04:09.047157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.047132    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f732077f-5bbf7898ff-jjg5h"]
Nov 03 12:04:09.682199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.682161    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" path="/var/lib/kubelet/pods/66fb29bb-4a43-4af4-b26a-f250e3d59be1/volumes"
Nov 03 12:04:09.773741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.773705    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:09.869408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.869343    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\" (UniqueName: \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\") pod \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") "
Nov 03 12:04:09.869605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.869464    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\") pod \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") "
Nov 03 12:04:09.869605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.869525    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\") pod \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\" (UID: \"cc4b1928-46b0-415e-9ed3-3579fcb2b559\") "
Nov 03 12:04:09.869933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.869906    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca" pod "cc4b1928-46b0-415e-9ed3-3579fcb2b559" (UID: "cc4b1928-46b0-415e-9ed3-3579fcb2b559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:09.871560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.871528    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8" pod "cc4b1928-46b0-415e-9ed3-3579fcb2b559" (UID: "cc4b1928-46b0-415e-9ed3-3579fcb2b559"). InnerVolumeSpecName "kube-api-access-jh5z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:09.871649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.871615    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca" pod "cc4b1928-46b0-415e-9ed3-3579fcb2b559" (UID: "cc4b1928-46b0-415e-9ed3-3579fcb2b559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:09.970674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.970582    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-openshift-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:09.970674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.970615    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh5z8\" (UniqueName: \"kubernetes.io/projected/cc4b1928-46b0-415e-9ed3-3579fcb2b559-kube-api-access-jh5z8\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:09.970674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:09.970630    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc4b1928-46b0-415e-9ed3-3579fcb2b559-maas-api-xks-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:10.645866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:10.645832    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7" event={"ID":"cc4b1928-46b0-415e-9ed3-3579fcb2b559","Type":"ContainerDied","Data":"f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b"}
Nov 03 12:04:10.645866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:10.645852    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"
Nov 03 12:04:10.646076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:10.645880    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f447250943257f24a938cd3ea377aed3af152ecdf188567bd64817c86db4048b"
Nov 03 12:04:11.143214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:11.143179    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:11.306267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:11.306240    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:11.460490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:11.460362    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:12.447557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:12.447522    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:13.120454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:13.120410    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:13.439744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:13.439704    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:13.713718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:13.713631    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:14.296968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:14.296940    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:14.512298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:14.512261    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:15.142366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:15.142331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:15.319479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:15.319442    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:16.727031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:16.726996    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:16.875934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:16.875890    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:18.815798 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:18.815764    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:19.600804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:19.600770    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:19.902146 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:19.902112    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:20.073654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:20.073619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:22.696008 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:22.695977    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:22.868540 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:22.868508    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:23.215269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:23.215221    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:23.524039 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:23.523961    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:04:23.702454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:23.702407    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:23.854619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:23.854491    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:24.996396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:24.996310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:26.471940 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:26.471904    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:04:28.524911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.524845    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" podUID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" containerName="payload-processing" containerID="cri-o://78cab4cae5c0b240aa76b7268bba39e8e5a51a6e0f989d4ec116e14d3db1de2e" gracePeriod=25
Nov 03 12:04:28.756420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.756387    5784 generic.go:356] "Generic (PLEG): container finished" podID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" containerID="78cab4cae5c0b240aa76b7268bba39e8e5a51a6e0f989d4ec116e14d3db1de2e" exitCode=0
Nov 03 12:04:28.756573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.756417    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" event={"ID":"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7","Type":"ContainerDied","Data":"78cab4cae5c0b240aa76b7268bba39e8e5a51a6e0f989d4ec116e14d3db1de2e"}
Nov 03 12:04:28.780430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.780367    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:04:28.859364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.859331    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\" (UniqueName: \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\") pod \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") "
Nov 03 12:04:28.859556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.859371    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\") pod \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\" (UID: \"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7\") "
Nov 03 12:04:28.859819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.859794    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume" pod "bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" (UID: "bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:28.861486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.861461    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld" pod "bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" (UID: "bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7"). InnerVolumeSpecName "kube-api-access-kw9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:28.960690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.960655    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw9ld\" (UniqueName: \"kubernetes.io/projected/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-kube-api-access-kw9ld\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:28.960690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:28.960684    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:29.572285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.572251    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:29.722405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.722361    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:29.765967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.765939    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"
Nov 03 12:04:29.766155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.765978    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7" event={"ID":"bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7","Type":"ContainerDied","Data":"2fc780261769290de1f54f510ab96e29ec585fea119fc2ca7cfdd064c2b57fe9"}
Nov 03 12:04:29.766155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.766050    5784 scope.go:122] "RemoveContainer" containerID="78cab4cae5c0b240aa76b7268bba39e8e5a51a6e0f989d4ec116e14d3db1de2e"
Nov 03 12:04:29.882846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.882814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:29.921539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.921501    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:04:29.925018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:29.924982    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1319e7ef-7f67884c45-h98m7"]
Nov 03 12:04:30.034730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:30.034698    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:30.629786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:30.629750    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:30.876196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:30.876161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:31.633854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:31.633822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:31.683462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:31.683433    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" path="/var/lib/kubelet/pods/bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7/volumes"
Nov 03 12:04:31.800171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:31.800138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:32.433680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:32.433637    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:32.603100 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:32.593826    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:32.766812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:32.766723    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:32.877875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:32.877835    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"]
Nov 03 12:04:32.923997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:32.923948    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-30e6ec98-c22r7"]
Nov 03 12:04:33.038641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.038538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:33.209705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.209649    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:33.408602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.408550    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:33.591977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.591935    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:33.681266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.681190    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc4b1928-46b0-415e-9ed3-3579fcb2b559" path="/var/lib/kubelet/pods/cc4b1928-46b0-415e-9ed3-3579fcb2b559/volumes"
Nov 03 12:04:33.770226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.770195    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:33.932616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:33.932528    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:35.354760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:35.354723    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:35.537224 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:35.537178    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:36.616731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:36.616681    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:37.067498 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:37.067459    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:37.282768 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:37.282741    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:37.450948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:37.450913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:37.995296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:37.995265    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:38.174987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:38.174953    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:38.357960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:38.357876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:38.579406 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:38.579345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:39.707041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:39.707010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:39.885886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:39.885854    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:40.169558 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.169525    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:40.343921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.343888    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:40.703141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.703109    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:04:40.703336 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.703314    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" podUID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" containerName="maas-api" containerID="cri-o://a2a5b986aa21d8ee9e3a238ae381eb329bd575cba1e2312b14db8f05d17a75dd" gracePeriod=30
Nov 03 12:04:40.834059 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.834030    5784 generic.go:356] "Generic (PLEG): container finished" podID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" containerID="a2a5b986aa21d8ee9e3a238ae381eb329bd575cba1e2312b14db8f05d17a75dd" exitCode=0
Nov 03 12:04:40.834536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.834083    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" event={"ID":"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa","Type":"ContainerDied","Data":"a2a5b986aa21d8ee9e3a238ae381eb329bd575cba1e2312b14db8f05d17a75dd"}
Nov 03 12:04:40.848117 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.848084    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:40.964559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:40.964496    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:04:41.077809 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.077776    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") pod \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") "
Nov 03 12:04:41.078001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.077938    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\") pod \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") "
Nov 03 12:04:41.078001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.077976    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\" (UniqueName: \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\") pod \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") "
Nov 03 12:04:41.078140 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.078024    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\") pod \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\" (UID: \"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa\") "
Nov 03 12:04:41.078636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.078603    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca" pod "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" (UID: "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:04:41.081092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.081063    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh" pod "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" (UID: "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa"). InnerVolumeSpecName "kube-api-access-55pzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:04:41.081092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.081080    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs" pod "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" (UID: "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:41.083799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.083768    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls" pod "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" (UID: "6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:04:41.179009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.178970    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:41.179009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.179005    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pzh\" (UniqueName: \"kubernetes.io/projected/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-kube-api-access-55pzh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:41.179266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.179019    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:41.179266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.179034    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:04:41.841800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.841768    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l" event={"ID":"6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa","Type":"ContainerDied","Data":"fadbaf6fc584b81efa22c769839bae41907b2a9d240ba31dd53bde799f4c3cf1"}
Nov 03 12:04:41.842332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.841805    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"
Nov 03 12:04:41.842332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.841828    5784 scope.go:122] "RemoveContainer" containerID="a2a5b986aa21d8ee9e3a238ae381eb329bd575cba1e2312b14db8f05d17a75dd"
Nov 03 12:04:41.994329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:41.994248    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:04:42.040286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:42.040245    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-9ab28c7a-6ff68b6c7-c8n7l"]
Nov 03 12:04:42.663793 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:42.663752    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:43.070412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:43.070291    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:43.424638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:43.424599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:43.680943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:43.680861    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" path="/var/lib/kubelet/pods/6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa/volumes"
Nov 03 12:04:43.734781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:43.734741    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:44.553994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:44.553962    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:44.739047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:44.739000    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:46.082727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:46.082693    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:46.990368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:46.990337    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:47.139331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.139285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:47.155557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.155528    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 12:04:47.156060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156043    5784 cpu_manager.go:414] "containerMap: removing container" podUID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" containerName="payload-processing"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156062    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" containerName="payload-processing"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156086    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" containerName="payload-processing"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156094    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" containerName="payload-processing"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156104    5784 cpu_manager.go:414] "containerMap: removing container" podUID="cc4b1928-46b0-415e-9ed3-3579fcb2b559" containerName="revoke-keys"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156112    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc4b1928-46b0-415e-9ed3-3579fcb2b559" containerName="revoke-keys"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156131    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" containerName="maas-api"
Nov 03 12:04:47.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156138    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" containerName="maas-api"
Nov 03 12:04:47.156569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156258    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fb29bb-4a43-4af4-b26a-f250e3d59be1" containerName="payload-processing"
Nov 03 12:04:47.156569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156274    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdbd6c92-5d8a-47b3-9f5a-a13fd23ec7f7" containerName="payload-processing"
Nov 03 12:04:47.156569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156284    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc4b1928-46b0-415e-9ed3-3579fcb2b559" containerName="revoke-keys"
Nov 03 12:04:47.156569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.156294    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e094aa8-fbf9-4c6e-9fb0-142bcb0d3daa" containerName="maas-api"
Nov 03 12:04:47.159856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.159835    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.238240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.238199    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9xb\" (UniqueName: \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.238456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.238333    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.238456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.238423    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.238584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.238460    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.257889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.257606    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-8d2c7c-serving-cert\""
Nov 03 12:04:47.257889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.257667    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 12:04:47.257889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.257724    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vv49\""
Nov 03 12:04:47.261523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.261499    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 12:04:47.338994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.338959    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.338994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.339000    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.339247 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:47.339127    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-serving-cert: secret "maas-api-e2e-worker-w4-8d2c7c-serving-cert" not found
Nov 03 12:04:47.339247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.339137    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9xb\" (UniqueName: \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.339247 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:47.339190    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls podName:eb844ff7-5348-4de3-aa50-82fdef7539f7 nodeName:}" failed. No retries permitted until 2026-11-03 12:04:47.839168903 +0000 UTC m=+794.281371188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls") pod "maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" (UID: "eb844ff7-5348-4de3-aa50-82fdef7539f7") : secret "maas-api-e2e-worker-w4-8d2c7c-serving-cert" not found
Nov 03 12:04:47.339439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.339415    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.339857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.339831    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.342036 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.342012    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.394321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.394289    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9xb\" (UniqueName: \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.412002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.411973    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:47.606463 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.606373    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:47.756365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.756331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:47.844919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.844881    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.847391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.847353    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") pod \"maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:47.950521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:47.950491    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:48.074009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:48.073972    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:48.109323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:48.109292    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:48.442086 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:48.442063    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 12:04:48.444443 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:04:48.444413    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb844ff7_5348_4de3_aa50_82fdef7539f7.slice/crio-e07df703af2ce4b677cbbea88ad7b28a2b7777fe30b0c09054e734e96833c075 WatchSource:0}: Error finding container e07df703af2ce4b677cbbea88ad7b28a2b7777fe30b0c09054e734e96833c075: Status 404 returned error can't find the container with id e07df703af2ce4b677cbbea88ad7b28a2b7777fe30b0c09054e734e96833c075
Nov 03 12:04:48.895237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:48.895204    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" event={"ID":"eb844ff7-5348-4de3-aa50-82fdef7539f7","Type":"ContainerStarted","Data":"e07df703af2ce4b677cbbea88ad7b28a2b7777fe30b0c09054e734e96833c075"}
Nov 03 12:04:49.503017 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:49.502987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:49.904198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:49.904159    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" event={"ID":"eb844ff7-5348-4de3-aa50-82fdef7539f7","Type":"ContainerStarted","Data":"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"}
Nov 03 12:04:49.904411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:49.904264    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:50.013368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:50.013314    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" podStartSLOduration=2.093881914 podStartE2EDuration="3.013299821s" totalImagesPullingTime="919.417907ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:04:50.012855441 +0000 UTC m=+796.455057743" watchObservedRunningTime="2026-11-03 12:04:50.013299821 +0000 UTC m=+796.455502125"
Nov 03 12:04:50.015996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:50.015960    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:50.193818 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:50.193732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:50.377111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:50.377059    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:52.255581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:52.255549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:55.250332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:55.250298    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:55.915395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:55.915355    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 12:04:56.478515 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:56.478479    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:04:56.870442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:56.870335    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 12:04:56.874372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:56.874351    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:56.925107 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:56.925066    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9893f0-serving-cert\""
Nov 03 12:04:56.973486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:56.973450    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 12:04:57.036986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.036956    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.037159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.036999    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.037159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.037080    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.037159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.037115    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjrg9\" (UniqueName: \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.074454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.074417    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:57.138226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.138193    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjrg9\" (UniqueName: \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.138439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.138266    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.138439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.138301    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.138439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.138358    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.138439 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:57.138392    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-serving-cert: secret "maas-api-e2e-worker-w0-9893f0-serving-cert" not found
Nov 03 12:04:57.138605 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:04:57.138474    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls podName:d30341b2-3f0e-4970-ac6c-3f2948df548f nodeName:}" failed. No retries permitted until 2026-11-03 12:04:57.638453683 +0000 UTC m=+804.080655980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls") pod "maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" (UID: "d30341b2-3f0e-4970-ac6c-3f2948df548f") : secret "maas-api-e2e-worker-w0-9893f0-serving-cert" not found
Nov 03 12:04:57.138995 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.138973    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.140838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.140810    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.195610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.195575    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjrg9\" (UniqueName: \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.277619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.277581    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:57.427313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.427222    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:57.643542 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.643507    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.646133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.646103    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.787411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.787292    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:04:57.954371 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:57.954337    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:58.122325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:58.122242    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:04:58.154060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:58.154030    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 12:04:58.155444 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:04:58.155417    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30341b2_3f0e_4970_ac6c_3f2948df548f.slice/crio-1ab1d327354ce797fe7f9f3379ac6ef059339fc8dfe65b78bd2532b54d1167e5 WatchSource:0}: Error finding container 1ab1d327354ce797fe7f9f3379ac6ef059339fc8dfe65b78bd2532b54d1167e5: Status 404 returned error can't find the container with id 1ab1d327354ce797fe7f9f3379ac6ef059339fc8dfe65b78bd2532b54d1167e5
Nov 03 12:04:58.960448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:58.960409    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" event={"ID":"d30341b2-3f0e-4970-ac6c-3f2948df548f","Type":"ContainerStarted","Data":"1ab1d327354ce797fe7f9f3379ac6ef059339fc8dfe65b78bd2532b54d1167e5"}
Nov 03 12:04:59.967517 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:59.967482    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" event={"ID":"d30341b2-3f0e-4970-ac6c-3f2948df548f","Type":"ContainerStarted","Data":"74c8b8828d73e3ca9e4b9c772b34bcdfab68028772a87b9e074f0f7903072ddf"}
Nov 03 12:04:59.968003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:04:59.967548    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:05:00.074257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:00.074062    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" podStartSLOduration=3.248026684 podStartE2EDuration="4.074043946s" totalImagesPullingTime="826.017262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:00.073455402 +0000 UTC m=+806.515657706" watchObservedRunningTime="2026-11-03 12:05:00.074043946 +0000 UTC m=+806.516246270"
Nov 03 12:05:02.080063 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:02.080018    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:02.271924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:02.271880    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:02.425948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:02.425917    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:03.408247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.408213    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:05:03.412304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.412282    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498428    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498481    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498625    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498709    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498761    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498824    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498850    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.498945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498928    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.499001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.498989    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpk6\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.510653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.510627    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:05:03.534767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.534737    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8d2c7c-openshift-default-dockercfg-tjwjb\""
Nov 03 12:05:03.600471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600433    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600500    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600530    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600578    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600621    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpk6\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600662    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600728    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600752    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.600941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600785    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.601098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600959    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.601098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.600997    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.601098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.601026    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.601253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.601169    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.601426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.601400    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.602945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.602924    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.603161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.603142    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.653333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.653298    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.653630 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.653605    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpk6\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:03.728201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:03.728103    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:04.151892 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.151855    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:05:04.156693 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:05:04.156662    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08c6341_545b_4f74_ad70_e93fccf83301.slice/crio-1ff551c5c4d8e6efaf25a99153440c12abb5fd730917962d05fe3ecc9354991e WatchSource:0}: Error finding container 1ff551c5c4d8e6efaf25a99153440c12abb5fd730917962d05fe3ecc9354991e: Status 404 returned error can't find the container with id 1ff551c5c4d8e6efaf25a99153440c12abb5fd730917962d05fe3ecc9354991e
Nov 03 12:05:04.159300 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.159265    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:04.159400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.159338    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:04.159444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.159410    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:04.997193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.997159    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" event={"ID":"d08c6341-545b-4f74-ad70-e93fccf83301","Type":"ContainerStarted","Data":"aace5f634c61a618aa6c3f72fdd6ddac54f0d3a023640c7b02408b41b1a01322"}
Nov 03 12:05:04.997697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:04.997209    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" event={"ID":"d08c6341-545b-4f74-ad70-e93fccf83301","Type":"ContainerStarted","Data":"1ff551c5c4d8e6efaf25a99153440c12abb5fd730917962d05fe3ecc9354991e"}
Nov 03 12:05:05.160399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:05.154761    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podStartSLOduration=2.154741511 podStartE2EDuration="2.154741511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:05.106322595 +0000 UTC m=+811.548524901" watchObservedRunningTime="2026-11-03 12:05:05.154741511 +0000 UTC m=+811.596943822"
Nov 03 12:05:05.731425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:05.731356    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:05.734860 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:05.734829    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:05:05.735065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:05.735037    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 12:05:05.977780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:05.977753    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 12:05:06.733401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:06.732796    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:05:06.733401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:06.732859    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 12:05:06.883194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:06.883162    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:07.735854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:07.735810    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:05:07.736495 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:07.736459    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 12:05:08.732527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:08.732488    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:05:08.732806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:08.732778    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 12:05:09.732840 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:09.732800    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 12:05:09.733475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:09.733443    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 12:05:10.733162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:10.733131    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:10.733654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:10.733515    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:10.734103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:10.734081    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:05:11.312345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:11.312314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:15.467795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.467762    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 12:05:15.471575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.471555    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.526082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.526049    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 12:05:15.553198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.553170    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.553362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.553210    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.553362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.553284    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxph9\" (UniqueName: \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.553362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.553331    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.568633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.568605    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5951bc-serving-cert\""
Nov 03 12:05:15.654202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.654161    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.654413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.654266    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.654413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.654307    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.654413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.654357    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxph9\" (UniqueName: \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.654592 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:05:15.654481    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-serving-cert: secret "maas-api-e2e-worker-w1-5951bc-serving-cert" not found
Nov 03 12:05:15.654592 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:05:15.654552    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls podName:708f97d2-541f-4676-a00d-db59163b08bc nodeName:}" failed. No retries permitted until 2026-11-03 12:05:16.154530722 +0000 UTC m=+822.596733024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls") pod "maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" (UID: "708f97d2-541f-4676-a00d-db59163b08bc") : secret "maas-api-e2e-worker-w1-5951bc-serving-cert" not found
Nov 03 12:05:15.655087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.655066    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.656692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.656667    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.682085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.682051    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:15.710326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.710287    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxph9\" (UniqueName: \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:15.844885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:15.844766    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:16.004297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:16.004261    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:16.160343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:16.160310    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:16.162896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:16.162863    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:16.385442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:16.385398    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:16.747633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:16.747605    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 12:05:16.749482 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:05:16.749451    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708f97d2_541f_4676_a00d_db59163b08bc.slice/crio-9516740dc395087b2708524597a9118ea96e3fec18b705f649754a16c4af88ad WatchSource:0}: Error finding container 9516740dc395087b2708524597a9118ea96e3fec18b705f649754a16c4af88ad: Status 404 returned error can't find the container with id 9516740dc395087b2708524597a9118ea96e3fec18b705f649754a16c4af88ad
Nov 03 12:05:17.101464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:17.101361    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" event={"ID":"708f97d2-541f-4676-a00d-db59163b08bc","Type":"ContainerStarted","Data":"9516740dc395087b2708524597a9118ea96e3fec18b705f649754a16c4af88ad"}
Nov 03 12:05:17.367045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:17.366965    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:17.947870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:17.947832    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:18.116364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:18.115631    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" event={"ID":"708f97d2-541f-4676-a00d-db59163b08bc","Type":"ContainerStarted","Data":"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"}
Nov 03 12:05:18.116364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:18.116046    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:18.144746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:18.144684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:18.224895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:18.224776    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" podStartSLOduration=2.412018259 podStartE2EDuration="3.224760202s" totalImagesPullingTime="812.741943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:18.223258518 +0000 UTC m=+824.665460851" watchObservedRunningTime="2026-11-03 12:05:18.224760202 +0000 UTC m=+824.666962501"
Nov 03 12:05:20.520801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.520768    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:20.678194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.678159    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:05:20.682275 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.682252    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.731422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.731365    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bb1b51-openshift-default-dockercfg-b9g7k\""
Nov 03 12:05:20.806433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806343    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806396    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806508    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806554    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806604    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806630    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806672    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806728    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58kb\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.806769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.806760    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.869254 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.869217    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:05:20.907668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907640    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.907843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907687    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.907843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907710    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.907843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907734    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.907843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907779    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.907843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907818    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907852    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907878    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.907920    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58kb\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908281 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.908153    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908281 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.908227    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908406 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.908306    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.908416    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.908535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.908516    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.910131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.910110    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.910484 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.910462    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.915816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.915778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:20.961238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.961205    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.961692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.961667    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58kb\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:20.995242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:20.995214    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:21.458517 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:21.458488    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:05:21.459859 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:05:21.459831    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f5c72c_5d6b_4a81_b0d9_edde64c4b3d7.slice/crio-c1edd01291a8e9193f243390c656f9685732b67d226d06a55c9b5523ef189e13 WatchSource:0}: Error finding container c1edd01291a8e9193f243390c656f9685732b67d226d06a55c9b5523ef189e13: Status 404 returned error can't find the container with id c1edd01291a8e9193f243390c656f9685732b67d226d06a55c9b5523ef189e13
Nov 03 12:05:21.462274 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:21.462237    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:21.462353 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:21.462309    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:21.462353 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:21.462336    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:22.150808 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.150767    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" event={"ID":"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7","Type":"ContainerStarted","Data":"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"}
Nov 03 12:05:22.151213 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.150826    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" event={"ID":"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7","Type":"ContainerStarted","Data":"c1edd01291a8e9193f243390c656f9685732b67d226d06a55c9b5523ef189e13"}
Nov 03 12:05:22.353857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.353800    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podStartSLOduration=2.353780528 podStartE2EDuration="2.353780528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:22.306628689 +0000 UTC m=+828.748831021" watchObservedRunningTime="2026-11-03 12:05:22.353780528 +0000 UTC m=+828.795982833"
Nov 03 12:05:22.996228 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.996189    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:22.997734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.997709    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 12:05:22.997851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:22.997757    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 12:05:23.996042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:23.996007    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 12:05:23.996469 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:23.996064    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 12:05:24.143999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:24.143965    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 12:05:24.996172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:24.996139    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 12:05:24.996557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:24.996192    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 12:05:25.141809 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:25.141774    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:25.995792 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:25.995754    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 12:05:25.995977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:25.995813    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 12:05:26.471830 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:05:26.471793    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:05:26.996190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:26.996156    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 12:05:26.996419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:26.996219    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 12:05:28.000891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:28.000860    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:28.001356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:28.001139    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:28.001888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:28.001871    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:05:28.220372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:28.220342    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:29.628559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:29.628528    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:33.389392 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:33.389325    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:34.024396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:34.024326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:34.273991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:34.273958    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:34.452348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:34.452297    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:39.126973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:39.126941    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:39.933638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:39.933566    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:40.544445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:40.544415    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:40.696500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:40.696469    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:43.638610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:43.638576    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:45.298245 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:45.298207    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:48.019928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.019898    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:48.684926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.684889    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:05:48.688906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.688884    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.770320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.770274    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:05:48.830617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830577    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830621    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830689    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830846    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830889    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830910    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.830965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.830935    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.831060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.831012    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.831060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.831031    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpdfc\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932056    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932098    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932122    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932171    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932200    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpdfc\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932232    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932256    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932286    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932457    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932586    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932805    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.932946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.932921    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.933046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.933014    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.933175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.933030    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.934760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.934735    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.934956 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.934913    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.985630 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.985597    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:48.985786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:48.985720    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpdfc\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:49.002743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:49.002709    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:49.423814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:49.423775    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:05:49.459918 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:05:49.459888    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe8b6db7_8a8f_474d_a706_a2613b836308.slice/crio-c794e207d894449053955d72d34b82da6ee19096ec32c24a505b857405a56799 WatchSource:0}: Error finding container c794e207d894449053955d72d34b82da6ee19096ec32c24a505b857405a56799: Status 404 returned error can't find the container with id c794e207d894449053955d72d34b82da6ee19096ec32c24a505b857405a56799
Nov 03 12:05:49.461990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:49.461956    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:49.462072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:49.462025    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:49.462072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:49.462056    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:05:50.347348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:50.347306    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" event={"ID":"fe8b6db7-8a8f-474d-a706-a2613b836308","Type":"ContainerStarted","Data":"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"}
Nov 03 12:05:50.347576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:50.347359    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" event={"ID":"fe8b6db7-8a8f-474d-a706-a2613b836308","Type":"ContainerStarted","Data":"c794e207d894449053955d72d34b82da6ee19096ec32c24a505b857405a56799"}
Nov 03 12:05:50.500432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:50.500364    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podStartSLOduration=2.500350029 podStartE2EDuration="2.500350029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:05:50.499844726 +0000 UTC m=+856.942047057" watchObservedRunningTime="2026-11-03 12:05:50.500350029 +0000 UTC m=+856.942552333"
Nov 03 12:05:50.767296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:50.767255    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:51.003240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:51.003202    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:51.004837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:51.004814    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 12:05:51.004944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:51.004860    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 12:05:51.469316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:51.469286    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:51.670731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:51.670693    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:52.003271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:52.003238    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 12:05:52.003444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:52.003301    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 12:05:53.003503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:53.003468    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 12:05:53.003893 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:53.003544    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 12:05:53.056986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:53.056950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:53.992180 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:53.992143    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:05:54.003165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:54.003137    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 12:05:54.003321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:54.003186    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 12:05:55.004001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:55.003962    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 12:05:55.004507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:55.004055    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 12:05:56.007736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:56.007705    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:56.008155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:56.007972    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:56.008702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:56.008687    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:05:56.470736 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:05:56.470698    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:05:57.515703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:05:57.515665    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:02.117861 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:02.117821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:05.342546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:05.342512    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:06.515239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:06.515208    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:09.747894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:09.747855    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:09.904733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:09.904684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:10.916091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:10.916062    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:15.331893 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:15.331855    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:19.820488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:19.820445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:20.261386 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.261337    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 12:06:20.265148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.265127    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.362364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.362333    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"kube-root-ca.crt\""
Nov 03 12:06:20.362608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.362442    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"openshift-service-ca.crt\""
Nov 03 12:06:20.362608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.362578    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"simulator-w1-5951bc-kserve-self-signed-certs\""
Nov 03 12:06:20.362608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.362585    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5951bc\"/\"default-dockercfg-lk5jw\""
Nov 03 12:06:20.365007 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.364989    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 12:06:20.365403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365384    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.365456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365418    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.365575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365556    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.365626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365601    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.365676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365639    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.365676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.365655    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8bs\" (UniqueName: \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.393967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.393934    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:20.466907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.466878    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.466916    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.466962    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.466979    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8bs\" (UniqueName: \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.467023    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.467074    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.467352    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.467399    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.467527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.467443    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.469425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.469402    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.469696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.469676    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.521759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.521622    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8bs\" (UniqueName: \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\") pod \"simulator-w1-5951bc-kserve-7679bd4d49-xhbrp\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") " pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.577535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.577494    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:20.952270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:20.952243    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 12:06:20.953736 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:06:20.953700    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f2e6d19_72f9_4327_8bd7_ea7bb992aebf.slice/crio-f217b57c61a875a41d118074e6134bcaa4aafbf2c80f909e93113b1e7633ab7a WatchSource:0}: Error finding container f217b57c61a875a41d118074e6134bcaa4aafbf2c80f909e93113b1e7633ab7a: Status 404 returned error can't find the container with id f217b57c61a875a41d118074e6134bcaa4aafbf2c80f909e93113b1e7633ab7a
Nov 03 12:06:21.555732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:21.555696    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" event={"ID":"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf","Type":"ContainerStarted","Data":"f217b57c61a875a41d118074e6134bcaa4aafbf2c80f909e93113b1e7633ab7a"}
Nov 03 12:06:23.574260 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:23.574172    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" event={"ID":"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf","Type":"ContainerStarted","Data":"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"}
Nov 03 12:06:23.578450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:23.578431    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:23.682934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:23.682866    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podStartSLOduration=1.474484313 podStartE2EDuration="3.68284639s" totalImagesPullingTime="2.208362077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:23.679909356 +0000 UTC m=+890.122111686" watchObservedRunningTime="2026-11-03 12:06:23.68284639 +0000 UTC m=+890.125048696"
Nov 03 12:06:24.313661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:24.313630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:26.473148 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:06:26.473112    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:06:27.602287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.602249    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 12:06:27.607123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.607094    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.660425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.660389    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 12:06:27.702609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.702573    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"default-dockercfg-n4q48\""
Nov 03 12:06:27.702609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.702594    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"kube-root-ca.crt\""
Nov 03 12:06:27.702609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.702600    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"openshift-service-ca.crt\""
Nov 03 12:06:27.702913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.702665    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8d2c7c\"/\"premium-w4-8d2c7c-kserve-self-signed-certs\""
Nov 03 12:06:27.723837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.723798    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:27.744282 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744251    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.744467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744305    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.744467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744421    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.744627 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744597    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.744679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744664    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfrz\" (UniqueName: \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.744733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.744705    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845342    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845401    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfrz\" (UniqueName: \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845598    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845651    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845784 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845684    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845784 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845717    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.845895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.845792    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.846058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.846034    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.846128 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.846110    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.848010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.847980    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.848220 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.848202    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.899903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.899839    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfrz\" (UniqueName: \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\") pod \"premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") " pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:27.921682 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:27.921640    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:28.295040 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.295008    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 12:06:28.296816 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:06:28.296788    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ab07899_0852_49b6_807b_84a24a08f5c9.slice/crio-db25feba59fe1618f65b5e9c999e9d77985a1b8a55a5b9ac35113ecaca19a8b7 WatchSource:0}: Error finding container db25feba59fe1618f65b5e9c999e9d77985a1b8a55a5b9ac35113ecaca19a8b7: Status 404 returned error can't find the container with id db25feba59fe1618f65b5e9c999e9d77985a1b8a55a5b9ac35113ecaca19a8b7
Nov 03 12:06:28.605706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.605621    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" event={"ID":"6ab07899-0852-49b6-807b-84a24a08f5c9","Type":"ContainerStarted","Data":"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"}
Nov 03 12:06:28.605706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.605659    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" event={"ID":"6ab07899-0852-49b6-807b-84a24a08f5c9","Type":"ContainerStarted","Data":"db25feba59fe1618f65b5e9c999e9d77985a1b8a55a5b9ac35113ecaca19a8b7"}
Nov 03 12:06:28.713885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.713820    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podStartSLOduration=1.713805534 podStartE2EDuration="1.713805534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:06:28.710598445 +0000 UTC m=+895.152800776" watchObservedRunningTime="2026-11-03 12:06:28.713805534 +0000 UTC m=+895.156007858"
Nov 03 12:06:28.816205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.816169    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:28.922046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:28.922012    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:30.578273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:30.578236    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:30.590989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:30.590966    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:30.635201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:30.635174    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:06:33.328064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:33.328031    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:33.814641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:33.814613    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:06:33.815646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:33.815625    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:06:37.784182 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:37.784138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:37.922369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:37.922336    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:37.940701 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:37.940668    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:38.698287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:38.698253    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:06:42.250295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:42.250191    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:42.409855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:42.409808    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:45.619089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:45.619059    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:46.794732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:46.794694    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:51.227188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:51.227148    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:55.672231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:06:55.672200    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:06:56.471500 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:06:56.471466    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:07:00.061578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:00.061538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:04.525723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:04.525687    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:08.706775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:08.706729    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:09.026911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:09.026830    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:13.526363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:13.526326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:18.063481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:18.063440    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:22.531811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:22.531753    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:22.704080 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:22.704044    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:26.472200 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:07:26.472169    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:07:27.024531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:27.024487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:29.428955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:29.428924    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:31.425892 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.425859    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:31.547565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.547526    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:07:31.552226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.552202    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.600676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.600649    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:07:31.650960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.650925    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:07:31.660293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.660259    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.660712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.660689    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glrng\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.660901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.660884    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661044 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661030    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661206 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661192    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661342    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661547 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661530    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661669    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.661952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.661935    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.762896    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.762931    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.762973    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763024    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763093    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glrng\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763128    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763162    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763212    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763249    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763414    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763753    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.763913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.763850    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.764033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.764008    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.764530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.764508    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.765525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.765504    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.765668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.765652    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.815604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.815571    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.815761 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.815741    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glrng\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:31.868825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:31.868788    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:32.290895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:32.290862    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:07:32.295242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:32.295197    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:32.295554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:32.295524    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:32.295739 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:32.295714    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:33.075257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.075217    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" event={"ID":"936e8f72-0cd7-4d59-abf4-9f74fba04ab2","Type":"ContainerStarted","Data":"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"}
Nov 03 12:07:33.075750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.075274    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" event={"ID":"936e8f72-0cd7-4d59-abf4-9f74fba04ab2","Type":"ContainerStarted","Data":"9cb552ea8e1ba796a0312751205e42706ee1599b21d2fface74b92b799717164"}
Nov 03 12:07:33.187256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.187204    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podStartSLOduration=2.187186854 podStartE2EDuration="2.187186854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:33.185334317 +0000 UTC m=+959.627536663" watchObservedRunningTime="2026-11-03 12:07:33.187186854 +0000 UTC m=+959.629389219"
Nov 03 12:07:33.869471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.869439    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:33.870974 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.870945    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:33.871041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:33.871014    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 12:07:34.869307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:34.869266    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:34.869733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:34.869342    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 12:07:35.870120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:35.870084    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:35.870602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:35.870139    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 12:07:36.044868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.044822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:36.666617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.666588    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:07:36.670696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.670673    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754160 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754116    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754160 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754163    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8zhb\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754289    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754330    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754365    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754467    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754517    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754543    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.754686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.754615    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.770320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.770281    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:07:36.856092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856057    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856109    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856157    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8zhb\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856190    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856227    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856250    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856307    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856330    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.856493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856352    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.857085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.856929    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.857085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.857024    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.857256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.857218    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.857398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.857359    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.857608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.857589    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.859266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.859207    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.859782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.859762    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.869735 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.869702    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:36.869830 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.869748    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 12:07:36.910754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.910714    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.911105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.910826    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8zhb\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:36.986896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:36.986795    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:37.446413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.440296    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:07:37.455201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.455163    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:37.455331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.455245    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:37.455331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.455287    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:07:37.870525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.870437    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 12:07:37.870525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:37.870508    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 12:07:38.159242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.159195    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" event={"ID":"f53c6021-1b1d-4c36-bec0-07bafc6c5446","Type":"ContainerStarted","Data":"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"}
Nov 03 12:07:38.159644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.159269    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" event={"ID":"f53c6021-1b1d-4c36-bec0-07bafc6c5446","Type":"ContainerStarted","Data":"a2d1e4b622f9540676d6d24aa4f9df5e3ef50d10ee44eb3aab2ba2db177e7ab0"}
Nov 03 12:07:38.274164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.274103    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podStartSLOduration=2.274089267 podStartE2EDuration="2.274089267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:07:38.271958725 +0000 UTC m=+964.714161050" watchObservedRunningTime="2026-11-03 12:07:38.274089267 +0000 UTC m=+964.716291572"
Nov 03 12:07:38.987658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.987607    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:38.989654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.989623    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:38.989811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:38.989673    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:39.910545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:39.910506    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:07:39.910973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:39.910568    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:07:39.987476 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:39.987430    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:39.987679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:39.987495    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:40.014159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.014124    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:40.015266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.015243    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:40.020400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.019649    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:07:40.656589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.656551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:40.987624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.987549    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:40.987624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:40.987600    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:41.987850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:41.987810    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:41.988246 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:41.987873    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:42.987928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:42.987893    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:42.988318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:42.987955    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:43.987996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:43.987959    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 12:07:43.988500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:43.988021    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 12:07:44.991810 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:44.991780    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:44.992291 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:44.992163    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:44.992773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:44.992753    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:07:45.114697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:45.114666    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:45.931428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:45.931394    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:49.536825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:49.536785    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:54.149171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:54.119299    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:07:56.478011 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:07:56.477971    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:07:58.635842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:07:58.635798    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:03.017672 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:03.017637    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:07.512235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:07.512198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:09.494346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:09.494308    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:11.992492 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:11.992406    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:12.147951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:12.147908    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:12.300211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:12.300124    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:15.933970 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:15.933930    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:16.518786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:16.518747    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:20.951890 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:20.951853    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:25.452526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:25.452478    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:26.473172 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:08:26.473132    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:08:29.326108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:29.326076    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:29.944623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:29.944591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:34.401695 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.401659    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:34.858632 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.858547    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:08:34.862778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.862757    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950150    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950189    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950212    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skl4\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950296    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950426    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950476    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950512    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950568    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.950749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.950705    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:34.961924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:34.961898    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:08:35.052165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052117    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052179    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052218    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052274    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052303    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052317    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052339    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skl4\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052370    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052453    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052607    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052686    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.052985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.052958    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.053127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.053059    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.053190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.053134    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.055149 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.055124    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.055349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.055334    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.105702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.105662    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.105886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.105862    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skl4\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\") pod \"e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") " pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.177344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.177314    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:35.590944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.590916    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:08:35.591365 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:08:35.591201    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586bf1ec_f2ea_47a9_861e_34e713b032d3.slice/crio-cbab09188a1ba2b0b8387800cf61d3de781d1cadc93e394286458c009910981f WatchSource:0}: Error finding container cbab09188a1ba2b0b8387800cf61d3de781d1cadc93e394286458c009910981f: Status 404 returned error can't find the container with id cbab09188a1ba2b0b8387800cf61d3de781d1cadc93e394286458c009910981f
Nov 03 12:08:35.593665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.593634    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:35.593747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.593697    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:35.593747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.593724    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:35.705778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.705737    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:08:35.709994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.709969    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.739319 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.739279    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" event={"ID":"586bf1ec-f2ea-47a9-861e-34e713b032d3","Type":"ContainerStarted","Data":"13553891f7cff89be04a69664e4f652b6f37cd173ed6b9923c7a5567793813d9"}
Nov 03 12:08:35.739515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.739349    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" event={"ID":"586bf1ec-f2ea-47a9-861e-34e713b032d3","Type":"ContainerStarted","Data":"cbab09188a1ba2b0b8387800cf61d3de781d1cadc93e394286458c009910981f"}
Nov 03 12:08:35.812406 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.812336    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:08:35.861955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.861860    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.861955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.861904    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.861955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.861922    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862243 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862121    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862243 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862160    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862323    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862560    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpvk\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862654    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.862841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.862822    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.925570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.925514    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podStartSLOduration=1.925499114 podStartE2EDuration="1.925499114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:35.923474785 +0000 UTC m=+1022.365677111" watchObservedRunningTime="2026-11-03 12:08:35.925499114 +0000 UTC m=+1022.367701420"
Nov 03 12:08:35.963562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963530    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963566    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963613    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963641    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963670    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963717    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpvk\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.963821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963788    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963849    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.963900    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.964094    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.964261    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.964504    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.964740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.964528    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.965030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.965006    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.966335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.966316    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:35.966692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:35.966671    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:36.016754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.016712    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:36.016999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.016975    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpvk\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\") pod \"maas-default-gateway-openshift-default-7f5667f86c-m5cbr\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:36.026651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.026620    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:36.177837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.177802    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:36.179324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.179296    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:36.179470 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.179348    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:36.460650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.460602    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:08:36.463746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.463713    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:36.463851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.463775    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:36.463851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.463807    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:36.749181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.749079    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" event={"ID":"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9","Type":"ContainerStarted","Data":"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"}
Nov 03 12:08:36.749181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.749151    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" event={"ID":"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9","Type":"ContainerStarted","Data":"a0606c5566c17925e1a79f1e32ed4963ce5ed480dc85b2adaa96cabc6f8124a6"}
Nov 03 12:08:36.860331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:36.860268    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podStartSLOduration=1.860245653 podStartE2EDuration="1.860245653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:36.855986433 +0000 UTC m=+1023.298188739" watchObservedRunningTime="2026-11-03 12:08:36.860245653 +0000 UTC m=+1023.302447960"
Nov 03 12:08:37.033466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:37.029441    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:37.033466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:37.029641    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:37.033466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:37.029696    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:37.178030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:37.177992    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:37.178203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:37.178048    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:38.034824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:38.034784    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:38.035475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:38.035442    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:38.178153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:38.178110    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:38.178420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:38.178371    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:39.032400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:39.030782    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:39.032400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:39.030861    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:39.150028 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:39.149989    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:39.177771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:39.177735    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:39.178058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:39.178028    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:40.036404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:40.033919    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:40.036404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:40.033982    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:40.177821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:40.177791    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:40.178315 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:40.178277    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:41.041402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:41.041230    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:41.041402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:41.041296    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:41.177842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:41.177801    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:41.178510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:41.178477    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:42.030407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:42.029820    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:42.030407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:42.029890    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:42.178653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:42.178620    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 12:08:42.179066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:42.178673    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 12:08:43.027947 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.027904    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:43.028136 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.027986    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:43.183413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.183365    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:43.183923 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.183894    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:43.184688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.184668    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:08:43.723745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:43.723696    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:44.028940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:44.028827    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 03 12:08:44.029257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:44.029221    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 03 12:08:46.070648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.070608    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:08:46.071247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.071216    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:08:46.226055 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.226017    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:46.227801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.227775    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:46.229153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.229132    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:08:46.276445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:46.275838    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:48.429926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:48.429883    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:50.807881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.807846    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:08:50.812849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.812827    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.907570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.907519    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:08:50.910975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.910948    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.910984    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911015    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911136    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911235    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911352    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911631    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911703    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjwt\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:50.911782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:50.911753    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013414    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjwt\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013458    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013484    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013499    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013524    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013547    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013626    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013681    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013765    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.013969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.013901    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.014179 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.014157    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.014244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.014198    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.014359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.014332    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.014529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.014475    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.016209 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.016186    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.016465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.016433    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.066192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.066123    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.066450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.066426    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjwt\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k6g4z\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.126184 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.126148    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:51.546876 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.546836    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:08:51.547997 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:08:51.547966    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe48a316_7d60_4ba9_81d5_cbd05632cd00.slice/crio-f11e51b6c01ac498c649a8db726100a3d5fa29fa0f10d4934a9885096fd2f20d WatchSource:0}: Error finding container f11e51b6c01ac498c649a8db726100a3d5fa29fa0f10d4934a9885096fd2f20d: Status 404 returned error can't find the container with id f11e51b6c01ac498c649a8db726100a3d5fa29fa0f10d4934a9885096fd2f20d
Nov 03 12:08:51.550761 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.550719    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:51.550878 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.550806    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:51.550878 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.550853    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:08:51.895049 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.895010    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" event={"ID":"be48a316-7d60-4ba9-81d5-cbd05632cd00","Type":"ContainerStarted","Data":"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"}
Nov 03 12:08:51.895485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:51.895073    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" event={"ID":"be48a316-7d60-4ba9-81d5-cbd05632cd00","Type":"ContainerStarted","Data":"f11e51b6c01ac498c649a8db726100a3d5fa29fa0f10d4934a9885096fd2f20d"}
Nov 03 12:08:52.003870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:52.003822    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podStartSLOduration=2.003807732 podStartE2EDuration="2.003807732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:08:52.001831056 +0000 UTC m=+1038.444033380" watchObservedRunningTime="2026-11-03 12:08:52.003807732 +0000 UTC m=+1038.446010079"
Nov 03 12:08:52.128855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:52.128820    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:52.129169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:52.128874    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:52.129169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:52.126856    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:08:52.888506 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:52.888474    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:53.132400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:53.127818    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:53.132400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:53.127877    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:54.137515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:54.130479    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:54.137515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:54.130541    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:55.136401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:55.135753    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:55.136401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:55.135822    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:56.134400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:56.133474    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:56.134400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:56.133539    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:56.474042 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:08:56.473963    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:08:57.129527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:57.129488    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:57.129828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:57.129796    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:57.342617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:57.342443    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:08:58.127465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:58.127422    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:58.127658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:58.127505    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:08:59.126969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:59.126920    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Nov 03 12:08:59.127493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:08:59.126993    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Nov 03 12:09:00.135983 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:00.135941    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:09:00.136455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:00.136269    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:09:00.137174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:00.137151    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:09:02.000921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:02.000872    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:05.868200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:05.868146    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:06.533596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:06.533558    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:10.933618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:10.933532    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:14.100128 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.100077    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 12:09:14.106076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.106053    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.158275 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.158241    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 12:09:14.178472 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178438    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czrxd\" (UniqueName: \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.178664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178587    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.178664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178614    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.178828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178678    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.178828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178702    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.178828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.178768    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.198944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.198916    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"kube-root-ca.crt\""
Nov 03 12:09:14.199473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.199432    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"default-dockercfg-87hfd\""
Nov 03 12:09:14.199473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.199445    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"openshift-service-ca.crt\""
Nov 03 12:09:14.200889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.200871    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bb1b51\"/\"distinct-w3-bb1b51-kserve-self-signed-certs\""
Nov 03 12:09:14.225984 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.225933    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:14.279669 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279633    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.279881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279698    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.279881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279722    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.279881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279772    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.279881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279838    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czrxd\" (UniqueName: \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.280108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.279941    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.280300 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.280276    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.282050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.282021    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.283320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.283026    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.288034 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.288007    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.292435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.292275    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.335992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.335957    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czrxd\" (UniqueName: \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\") pod \"distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") " pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.420104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.420065    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:14.855684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:14.855647    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 12:09:14.855971 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:09:14.855946    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7919092e_f3e8_491a_987c_303bca4a3486.slice/crio-4a88093f1385ccd0225ff2be493efc95dc005a0105e4ad7cfea7f79860fb2250 WatchSource:0}: Error finding container 4a88093f1385ccd0225ff2be493efc95dc005a0105e4ad7cfea7f79860fb2250: Status 404 returned error can't find the container with id 4a88093f1385ccd0225ff2be493efc95dc005a0105e4ad7cfea7f79860fb2250
Nov 03 12:09:15.128593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:15.128555    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" event={"ID":"7919092e-f3e8-491a-987c-303bca4a3486","Type":"ContainerStarted","Data":"fe17dbcb1fb65c8bed158fa32f3a46b628fa86d418054f93c06bc3a6cf4f01d7"}
Nov 03 12:09:15.128991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:15.128610    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" event={"ID":"7919092e-f3e8-491a-987c-303bca4a3486","Type":"ContainerStarted","Data":"4a88093f1385ccd0225ff2be493efc95dc005a0105e4ad7cfea7f79860fb2250"}
Nov 03 12:09:15.236178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:15.236122    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podStartSLOduration=1.236106073 podStartE2EDuration="1.236106073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:15.233707652 +0000 UTC m=+1061.675909958" watchObservedRunningTime="2026-11-03 12:09:15.236106073 +0000 UTC m=+1061.678308417"
Nov 03 12:09:15.409789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:15.409700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:15.420677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:15.420645    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:19.817477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:19.817441    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:24.224220 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:24.224177    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:24.420661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:24.420623    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:24.433846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:24.433809    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:25.216730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:25.216689    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:09:26.473337 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:09:26.473300    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:09:28.748401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:28.747989    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:33.133120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:33.133083    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:34.091073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:34.091038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:34.239137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:34.239100    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:37.138515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:37.138474    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:37.546646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:37.546549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:42.125328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:42.125256    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:44.786765 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:44.786734    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:44.931959 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:44.931918    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:45.080030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:45.079880    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:46.593238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:46.593203    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:48.487536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:48.487500    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:51.023812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:51.023758    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:52.672546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.672507    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:09:52.701724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.701692    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.845484 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.845444    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924117    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924174    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924231    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhw4k\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924269    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924315    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.924749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.924415    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.927207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.926674    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.927207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.926773    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:52.927207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:52.926826    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.028743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.028329    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhw4k\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.029050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.029026    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.029763 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.029631    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.030188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.030149    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.030329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.030311    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.030524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.030506    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.031173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.031155    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.032086 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.031944    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.032233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.032214    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.032751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.032732    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.033252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.033134    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.033537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.033519    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.034027 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.034008    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.034686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.034457    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.055732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.055696    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.059024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.056030    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.084799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.084725    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhw4k\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.126093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.125815    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\") pod \"e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") " pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:53.336035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:53.335686    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:54.459405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:54.444789    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:09:54.585759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:54.575227    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:09:54.585759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:54.575327    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:09:54.585759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:54.575394    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:09:55.208523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.208472    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.459906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.459632    5784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.710457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.708620    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:09:55.752914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.752870    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" event={"ID":"788a7a8c-6712-4e58-86c3-29c577a167d7","Type":"ContainerStarted","Data":"f28d497924927a80dbbe2a34578b18ba4b56383f9d4a5c5f1dc3d078feff5760"}
Nov 03 12:09:55.753102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.752942    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" event={"ID":"788a7a8c-6712-4e58-86c3-29c577a167d7","Type":"ContainerStarted","Data":"290c46ee2b497d6ccbcd7cb48095b01290404f7caf6ae4813f0c0b2a80ee56f1"}
Nov 03 12:09:55.898460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:55.898081    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podStartSLOduration=3.8980612580000003 podStartE2EDuration="3.898061258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:09:55.893661278 +0000 UTC m=+1102.335863582" watchObservedRunningTime="2026-11-03 12:09:55.898061258 +0000 UTC m=+1102.340263563"
Nov 03 12:09:56.247872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:56.247631    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:09:56.421431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:56.356018    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:09:56.542460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:56.532593    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:09:56.542460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:56.532703    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:09:56.732702 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:09:56.732640    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:09:57.339401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:57.338756    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:09:57.339401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:57.338815    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:09:58.337878 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:58.337833    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:09:58.338519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:58.338473    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:09:59.338823 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:59.338787    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:09:59.339486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:09:59.339440    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:10:00.340926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:00.340875    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:10:00.341364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:00.340947    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:10:00.946687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:00.946648    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:01.338954 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:01.338848    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:10:01.339253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:01.339210    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:10:02.336886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:02.336851    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 12:10:02.337285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:02.336911    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 12:10:03.340819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:03.340789    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:10:03.341297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:03.341187    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:10:03.341986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:03.341964    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:10:05.334973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:05.334939    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:05.491233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:05.491200    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:09.736047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:09.736011    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:14.230399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:14.229702    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:17.281594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:17.281550    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:10:17.282109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:17.281631    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:10:18.830283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:18.830249    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:22.208966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:22.208936    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:23.307736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:23.307699    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:26.472912 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:10:26.472873    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:10:27.721140 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:27.721106    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:32.114592 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.114534    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:32.703745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.703704    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:10:32.708282 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.708257    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.756750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.756721    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5951bc-openshift-default-dockercfg-bp9bz\""
Nov 03 12:10:32.788785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.788740    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.788984 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.788800    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.788984 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.788965    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789000    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789016    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxfv\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789143    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789260    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789314    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.789403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.789333    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.807946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.807910    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:10:32.890359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890322    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890418    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890453    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890491    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890516    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890562    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890580    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890598    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxfv\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890650    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.890934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890886    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.891117 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.890949    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.891183 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.891160    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.891240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.891190    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.891240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.891221    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.893343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.893321    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.893503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.893486    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.943778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.943740    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:32.943920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:32.943784    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxfv\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:33.023752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.023670    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:33.435531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.435499    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:10:33.436611 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:10:33.436585    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ddae72_8c9f_46ea_9136_35b3f84289ba.slice/crio-8f862446471a8cf105e0955bef41999d7220fd292ebf7ef515ce332c99f16317 WatchSource:0}: Error finding container 8f862446471a8cf105e0955bef41999d7220fd292ebf7ef515ce332c99f16317: Status 404 returned error can't find the container with id 8f862446471a8cf105e0955bef41999d7220fd292ebf7ef515ce332c99f16317
Nov 03 12:10:33.438816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.438781    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:10:33.438907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.438846    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:10:33.438907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.438874    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:10:33.899230 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:33.899199    5784 scope.go:122] "RemoveContainer" containerID="dc09140b5b241e69c0083bbd865283f1719473f515689912d849dd6c05e670de"
Nov 03 12:10:34.190535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:34.190447    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" event={"ID":"32ddae72-8c9f-46ea-9136-35b3f84289ba","Type":"ContainerStarted","Data":"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"}
Nov 03 12:10:34.190535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:34.190508    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" event={"ID":"32ddae72-8c9f-46ea-9136-35b3f84289ba","Type":"ContainerStarted","Data":"8f862446471a8cf105e0955bef41999d7220fd292ebf7ef515ce332c99f16317"}
Nov 03 12:10:34.392079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:34.392025    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podStartSLOduration=2.392010807 podStartE2EDuration="2.392010807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:10:34.345167459 +0000 UTC m=+1140.787369785" watchObservedRunningTime="2026-11-03 12:10:34.392010807 +0000 UTC m=+1140.834213147"
Nov 03 12:10:35.026400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:35.024687    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:35.033411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:35.032469    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:35.033411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:35.032527    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:36.027511 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:36.027470    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:36.028028 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:36.027544    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:36.441331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:36.441292    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:36.588239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:36.588203    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:37.024838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:37.024798    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:37.025165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:37.025135    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:38.030884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:38.030844    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:38.031521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:38.031488    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:39.034403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:39.033480    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:39.034403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:39.033551    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:40.024928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:40.024883    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 12:10:40.025148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:40.024952    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 12:10:41.029104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:41.029072    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:41.029712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:41.029507    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:41.030259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:41.030239    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:10:41.133226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:41.133194    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:45.514852 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:45.514809    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:46.902394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:46.902345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:49.996354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:49.996317    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:54.631305 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:54.631268    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:56.473326 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:10:56.473282    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:10:56.788093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:56.788015    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:58.933766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:58.933727    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 12:10:58.937888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:58.937865    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.003237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.000880    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 12:10:59.033358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.033305    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"openshift-service-ca.crt\""
Nov 03 12:10:59.033571 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.033435    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"kube-root-ca.crt\""
Nov 03 12:10:59.033800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.033780    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"distinct-w0-9893f0-kserve-self-signed-certs\""
Nov 03 12:10:59.033800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.033778    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9893f0\"/\"default-dockercfg-thlkn\""
Nov 03 12:10:59.050812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.050773    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:59.111214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111176    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.111431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111243    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.111431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111341    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.111431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111407    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trq6r\" (UniqueName: \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.111546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111491    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.111546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.111522    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.206122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.206034    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:10:59.212354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212325    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212371    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trq6r\" (UniqueName: \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212432    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212453    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212573    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212648    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212835    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.212991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.212968    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.213241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.213219    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.215334 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.215310    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.215455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.215310    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.265304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.265257    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trq6r\" (UniqueName: \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\") pod \"distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") " pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.550415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.550282    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:10:59.919704 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:10:59.919673    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabc14fde_2904_42ca_b96e_97107c23d314.slice/crio-4ea2691b65bd6b05b10496ca7043b939b169d5e04c9b0a27edeeca266be15588 WatchSource:0}: Error finding container 4ea2691b65bd6b05b10496ca7043b939b169d5e04c9b0a27edeeca266be15588: Status 404 returned error can't find the container with id 4ea2691b65bd6b05b10496ca7043b939b169d5e04c9b0a27edeeca266be15588
Nov 03 12:10:59.920903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:10:59.920871    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 12:11:00.399087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:00.399047    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" event={"ID":"abc14fde-2904-42ca-b96e-97107c23d314","Type":"ContainerStarted","Data":"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"}
Nov 03 12:11:00.399551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:00.399095    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" event={"ID":"abc14fde-2904-42ca-b96e-97107c23d314","Type":"ContainerStarted","Data":"4ea2691b65bd6b05b10496ca7043b939b169d5e04c9b0a27edeeca266be15588"}
Nov 03 12:11:00.507888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:00.507833    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podStartSLOduration=2.507818853 podStartE2EDuration="2.507818853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:00.504268488 +0000 UTC m=+1166.946470811" watchObservedRunningTime="2026-11-03 12:11:00.507818853 +0000 UTC m=+1166.950021158"
Nov 03 12:11:00.551163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:00.551126    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:11:02.800622 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:02.800582    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:03.628637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:03.628599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:08.239390 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:08.239350    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:09.551462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:09.551425    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:11:09.564365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:09.564337    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:11:10.491294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:10.491263    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:11:12.645054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:12.644977    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:17.127171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:17.127133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:18.066942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.066905    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:11:18.071079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.071058    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.172967    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173060    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173182    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbvx\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173268    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173351    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173449    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173495    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173572    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.173740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.173602    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.264081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.264043    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:11:18.275191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275118    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbvx\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275163    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275202    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275284    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275331    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275428    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275458    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275518    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275581    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.275827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.275805    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.276126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.276104    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.276273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.276124    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.276412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.276393    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.276525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.276510    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.278338 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.278316    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.278797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.278772    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.327966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.327891    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbvx\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.328119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.327977    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.388141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.388105    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:18.851801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.851765    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:11:18.853099 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:11:18.853069    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba5f4d9_ecab_43f8_93c4_78072c3912fe.slice/crio-4be0a96074ed301097d347361c78c89be695c5c525778b6a446137df0ea58a6a WatchSource:0}: Error finding container 4be0a96074ed301097d347361c78c89be695c5c525778b6a446137df0ea58a6a: Status 404 returned error can't find the container with id 4be0a96074ed301097d347361c78c89be695c5c525778b6a446137df0ea58a6a
Nov 03 12:11:18.855284 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.855253    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:18.855357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.855318    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:18.855357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:18.855346    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:19.564453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:19.564413    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" event={"ID":"0ba5f4d9-ecab-43f8-93c4-78072c3912fe","Type":"ContainerStarted","Data":"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"}
Nov 03 12:11:19.564856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:19.564470    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" event={"ID":"0ba5f4d9-ecab-43f8-93c4-78072c3912fe","Type":"ContainerStarted","Data":"4be0a96074ed301097d347361c78c89be695c5c525778b6a446137df0ea58a6a"}
Nov 03 12:11:19.790830 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:19.790774    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podStartSLOduration=1.7907587820000002 podStartE2EDuration="1.790758782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:19.787823483 +0000 UTC m=+1186.230025792" watchObservedRunningTime="2026-11-03 12:11:19.790758782 +0000 UTC m=+1186.232961145"
Nov 03 12:11:20.389097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:20.389054    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:20.390616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:20.390585    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:20.390734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:20.390645    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:21.388659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:21.388623    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:21.389052 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:21.388678    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:21.573757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:21.573721    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:22.388635 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:22.388589    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:22.388819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:22.388663    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:23.388619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:23.388581    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:23.388816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:23.388648    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:24.388466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:24.388428    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:24.388948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:24.388485    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:25.388650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:25.388617    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 12:11:25.389050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:25.388672    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 12:11:26.001714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:26.001678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:26.438477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:26.438447    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:26.440661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:26.440641    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:26.441781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:26.441763    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:11:27.225631 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:11:27.225585    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:11:29.046905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:29.046500    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:29.046905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:29.046585    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:31.027790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:31.027747    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:33.163497 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.163459    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:11:33.168100 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.168080    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.267759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.267726    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:11:33.287993 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.287958    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8bj\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288116    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288140    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288170    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288205    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288251    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288273    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288297    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.288370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.288318    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390056 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390024    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390111    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8bj\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390180    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390198    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390218    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390248    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390296    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390325    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390513    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390668    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390699    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390792    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.390915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.390900    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.391096 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.391065    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.392899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.392875    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.393095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.393075    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.442781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.442708    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.442913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.442837    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8bj\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\") pod \"e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") " pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.492604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.492575    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:33.869845 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.869759    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:11:33.871388 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.871345    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:11:33.904435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.904405    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:11:33.905291 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:11:33.905269    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f09b10e_f664_43bc_ab2c_ba3fa691dea9.slice/crio-0552cea753971ae53da7df1cb9eed210b343e6fc2b8c9da1645799eae78979d0 WatchSource:0}: Error finding container 0552cea753971ae53da7df1cb9eed210b343e6fc2b8c9da1645799eae78979d0: Status 404 returned error can't find the container with id 0552cea753971ae53da7df1cb9eed210b343e6fc2b8c9da1645799eae78979d0
Nov 03 12:11:33.907306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.907272    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:33.907429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.907332    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:33.907429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:33.907359    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:11:34.254451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.254417    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" event={"ID":"9f09b10e-f664-43bc-ab2c-ba3fa691dea9","Type":"ContainerStarted","Data":"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"}
Nov 03 12:11:34.254950 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.254470    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" event={"ID":"9f09b10e-f664-43bc-ab2c-ba3fa691dea9","Type":"ContainerStarted","Data":"0552cea753971ae53da7df1cb9eed210b343e6fc2b8c9da1645799eae78979d0"}
Nov 03 12:11:34.364015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.363964    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podStartSLOduration=1.363947707 podStartE2EDuration="1.363947707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:11:34.362304418 +0000 UTC m=+1200.804506746" watchObservedRunningTime="2026-11-03 12:11:34.363947707 +0000 UTC m=+1200.806150014"
Nov 03 12:11:34.493217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.493183    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:34.500397 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.498625    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:34.500397 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:34.498682    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:35.367689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:35.367658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:35.493295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:35.493259    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:35.493491 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:35.493318    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:36.493175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:36.493144    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:36.493576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:36.493194    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:37.493101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:37.493065    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:37.493268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:37.493132    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:38.493625 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:38.493591    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:38.494031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:38.493655    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:39.493153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:39.493115    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:39.493343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:39.493192    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:39.897482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:39.897449    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:40.493702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:40.493666    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 12:11:40.494033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:40.494003    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 12:11:42.511329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:42.511291    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:42.511812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:42.511366    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:43.552596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:43.552547    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:11:43.553236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:43.552634    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:11:43.599174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:43.599143    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:43.599616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:43.599581    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:43.600481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:43.600455    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:11:44.638081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:44.638049    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:47.230949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:47.230917    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:47.388678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:47.388628    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:49.017024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:49.016988    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:53.413324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:53.413288    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:54.768294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:54.768235    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:11:56.473116 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:11:56.473082    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:11:57.926983 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:11:57.926947    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:02.740052 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:02.740013    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:02.897435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:02.896907    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:07.471959 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:07.471912    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:11.924500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:11.924451    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:16.342666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:16.342616    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:20.557404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:20.550934    5784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:20.630401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:20.629520    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.217519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.217468    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.481581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.481477    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.583552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.583496    5784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.694404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.681512    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:21.694404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.682124    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:12:21.694404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:21.691889    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:22.261407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:22.255839    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:22.297404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:22.271700    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:12:22.371726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:22.371694    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:12:22.543859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:22.543777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:22.595358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:22.595328    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:12:26.471209 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:12:26.471167    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:12:27.126162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:27.126126    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:31.342234 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:31.342178    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:31.505823 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:31.505786    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:35.918852 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:35.918814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:38.150290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:38.150253    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:40.789411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:40.788938    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:41.387526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.387473    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.481552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.481464    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded"
Nov 03 12:12:41.599403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.598472    5784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.641460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.623599    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:12:41.709412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.696472    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:41.785410 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.784486    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:41.785410 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:41.784576    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.053676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.053230    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.053676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.053300    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.388733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.388677    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:12:42.389765 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.389738    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:12:42.532401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.506812    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:12:42.532401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.506882    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.624187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.624150    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 12:12:42.626923 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.626897    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:12:42.698298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.698206    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:12:42.699914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.699888    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:12:42.713401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:42.711976    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:12:43.398414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:43.398100    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:12:43.611882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:43.611306    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:12:43.614471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:43.613589    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:12:46.429061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:46.429021    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:47.543964 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:47.543934    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:50.128746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:50.128703    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:50.816595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:50.816563    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:51.982624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:51.982587    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:12:51.986816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:51.986796    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072271    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxlc4\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072317    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072540    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072598    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072633    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072731    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072765    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072871    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.072967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.072953    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.082740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.082710    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 12:12:52.086801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.086778    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:12:52.173912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.173876    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.173954    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174003    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxlc4\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174048    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174168    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174209    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174238    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174267    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174297    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174326    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174766    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174839    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174882    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.174975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.174936    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.176438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.176416    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.176650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.176630    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.226567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.226534    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxlc4\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.226702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.226534    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.302783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.302688    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:52.722420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.722392    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:12:52.722680 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:12:52.722657    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1de832_a360_4f45_beb5_ad9d1a191f9c.slice/crio-3338d23af9913c3c784f005076366c4875c67b77568af6515a4693d9fb86a8c0 WatchSource:0}: Error finding container 3338d23af9913c3c784f005076366c4875c67b77568af6515a4693d9fb86a8c0: Status 404 returned error can't find the container with id 3338d23af9913c3c784f005076366c4875c67b77568af6515a4693d9fb86a8c0
Nov 03 12:12:52.725099 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.725061    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:12:52.725192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.725127    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:12:52.725192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:52.725157    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:12:53.306008 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:53.305960    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" event={"ID":"ae1de832-a360-4f45-beb5-ad9d1a191f9c","Type":"ContainerStarted","Data":"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"}
Nov 03 12:12:53.306451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:53.306030    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" event={"ID":"ae1de832-a360-4f45-beb5-ad9d1a191f9c","Type":"ContainerStarted","Data":"3338d23af9913c3c784f005076366c4875c67b77568af6515a4693d9fb86a8c0"}
Nov 03 12:12:53.442204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:53.442132    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podStartSLOduration=2.442109583 podStartE2EDuration="2.442109583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:12:53.415335769 +0000 UTC m=+1279.857538075" watchObservedRunningTime="2026-11-03 12:12:53.442109583 +0000 UTC m=+1279.884311890"
Nov 03 12:12:54.303659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:54.303623    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:12:54.305429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:54.305401    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:54.305576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:54.305455    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:54.612357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:54.612266    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:55.322403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:55.315061    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:55.322403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:55.315135    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:55.348815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:55.348775    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:55.677164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:55.677128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:55.832218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:55.832186    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:56.303081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:56.303046    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:56.303301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:56.303104    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:56.476517 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:12:56.476473    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:12:57.304048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:57.304014    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:57.304590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:57.304091    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:58.303660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:58.303623    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:58.303834 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:58.303678    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:59.054161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:59.054119    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:59.219390 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:59.219329    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:12:59.304013 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:59.303981    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Nov 03 12:12:59.304228 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:59.304034    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Nov 03 12:12:59.403811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:12:59.403771    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:00.307901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:00.307868    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:13:00.308299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:00.308148    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:13:00.308856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:00.308840    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:13:00.583214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:00.583093    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:02.127131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:02.127093    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:02.296559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:02.296514    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:04.428978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:04.428941    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:05.145307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:05.145267    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:09.707106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:09.707069    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:13.698445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:13.698373    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:13.853152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:13.853111    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:14.142889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:14.142851    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:18.604417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:18.602012    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:23.004239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:23.004205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:26.470458 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:13:26.470419    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:13:27.146362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.146328    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:13:27.150498 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.150467    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.199952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.199920    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-69fd8043-serving-cert\""
Nov 03 12:13:27.251968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.251930    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:13:27.290757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.290719    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6bz8\" (UniqueName: \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.290944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.290766    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.290944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.290898    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.291098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.291078    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.393239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.393192    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.393456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.393275    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6bz8\" (UniqueName: \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.393456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.393298    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.393456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.393365    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.393947 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:13:27.393925    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-serving-cert: secret "maas-api-e2e-mt-69fd8043-serving-cert" not found
Nov 03 12:13:27.394128 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:13:27.393988    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls podName:81cfc45f-842f-4d4c-b316-4a22179cf697 nodeName:}" failed. No retries permitted until 2026-11-03 12:13:27.893966686 +0000 UTC m=+1314.336168969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls") pod "maas-api-e2e-mt-69fd8043-64675459d-4jqbf" (UID: "81cfc45f-842f-4d4c-b316-4a22179cf697") : secret "maas-api-e2e-mt-69fd8043-serving-cert" not found
Nov 03 12:13:27.396885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.396820    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.398259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.398232    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.448595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.448562    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6bz8\" (UniqueName: \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.899890 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.899846    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:27.902993 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:27.902966    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") pod \"maas-api-e2e-mt-69fd8043-64675459d-4jqbf\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:28.065877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.065841    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:28.439076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.439043    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:13:28.439519 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:13:28.439481    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cfc45f_842f_4d4c_b316_4a22179cf697.slice/crio-e97ee3a527d14f932d8a7ff972b5f3282ef5baedcad65a882642972664839474 WatchSource:0}: Error finding container e97ee3a527d14f932d8a7ff972b5f3282ef5baedcad65a882642972664839474: Status 404 returned error can't find the container with id e97ee3a527d14f932d8a7ff972b5f3282ef5baedcad65a882642972664839474
Nov 03 12:13:28.441030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.441010    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:13:28.770190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.770104    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" event={"ID":"81cfc45f-842f-4d4c-b316-4a22179cf697","Type":"ContainerStarted","Data":"e97ee3a527d14f932d8a7ff972b5f3282ef5baedcad65a882642972664839474"}
Nov 03 12:13:28.832668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.832620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:28.992716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:28.992681    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:29.779297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:29.779262    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" event={"ID":"81cfc45f-842f-4d4c-b316-4a22179cf697","Type":"ContainerStarted","Data":"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"}
Nov 03 12:13:29.779477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:29.779324    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:29.888665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:29.888603    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" podStartSLOduration=2.000525572 podStartE2EDuration="2.888584631s" totalImagesPullingTime="888.059059ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:29.885504524 +0000 UTC m=+1316.327706829" watchObservedRunningTime="2026-11-03 12:13:29.888584631 +0000 UTC m=+1316.330786937"
Nov 03 12:13:30.367111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.367070    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:13:30.371409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.371360    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.422569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.422532    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 12:13:30.422569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.422532    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 12:13:30.422794 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.422536    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 12:13:30.422794 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.422665    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rhbbs\""
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429597    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429653    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429739    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429793    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxz7z\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429875    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429898    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.430348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.429951    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.472690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.472658    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:13:30.530941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.530909    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.530941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.530948    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.530989    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.531089    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.531117    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.531154    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.531172    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxz7z\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.531801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.531772    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.533568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.533539    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.533664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.533580    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.533846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.533827    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.533892 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.533875    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.589292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.589210    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.589487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.589306    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxz7z\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bhj8s\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:30.687809 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:30.687775    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:31.109900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:31.109867    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:13:31.110320 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:13:31.110293    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6d3196_245a_456c_b905_82142206ab5e.slice/crio-bb03bb62bea65f15c2a54194b40528b64256b52abad7f6edc99740992e660c76 WatchSource:0}: Error finding container bb03bb62bea65f15c2a54194b40528b64256b52abad7f6edc99740992e660c76: Status 404 returned error can't find the container with id bb03bb62bea65f15c2a54194b40528b64256b52abad7f6edc99740992e660c76
Nov 03 12:13:31.795987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:31.795946    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" event={"ID":"5e6d3196-245a-456c-b905-82142206ab5e","Type":"ContainerStarted","Data":"bb03bb62bea65f15c2a54194b40528b64256b52abad7f6edc99740992e660c76"}
Nov 03 12:13:32.003831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:32.003795    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:34.137202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:34.137163    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:13:34.137714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:34.137416    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy" containerID="cri-o://a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50" gracePeriod=30
Nov 03 12:13:35.122263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:35.122225    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:35.286432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:35.286334    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:35.794564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:35.794541    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:13:35.794789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:35.794761    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:13:35.794838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:35.794822    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:13:36.555616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:36.555575    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:36.839810 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:36.839715    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" event={"ID":"5e6d3196-245a-456c-b905-82142206ab5e","Type":"ContainerStarted","Data":"761b33244b8c9daf0fc4fd3868528db2f2c0cfbafbb9d0d6cc5e4a8676540ced"}
Nov 03 12:13:36.840215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:36.840190    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:36.842324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:36.842295    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bhj8s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:13:36.842477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:36.842354    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:13:37.041224 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:37.041170    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" podStartSLOduration=2.35997639 podStartE2EDuration="7.041154354s" totalImagesPullingTime="4.681177964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:13:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:13:36.994237397 +0000 UTC m=+1323.436439727" watchObservedRunningTime="2026-11-03 12:13:37.041154354 +0000 UTC m=+1323.483356659"
Nov 03 12:13:37.848596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:37.848562    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:13:39.432742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.431735    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:39.453718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.453692    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:13:39.565211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565182    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565267    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565312    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565359    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565464    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565495    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:39.565578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565548    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565586    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565610    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:39.565743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565650    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565688    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\") pod \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\" (UID: \"936e8f72-0cd7-4d59-abf4-9f74fba04ab2\") "
Nov 03 12:13:39.565953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.565904    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:39.566470 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.566355    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.566595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.566513    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:39.566738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.566691    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.566738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.566717    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.566867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.566802    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:13:39.568003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.567982    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:13:39.568095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.568037    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:39.568197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.568178    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "kube-api-access-glrng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:13:39.568638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.568615    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy" pod "936e8f72-0cd7-4d59-abf4-9f74fba04ab2" (UID: "936e8f72-0cd7-4d59-abf4-9f74fba04ab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:13:39.595129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.595098    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:39.671461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671369    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.671461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671418    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.671461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671434    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.671461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671448    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.671704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671464    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glrng\" (UniqueName: \"kubernetes.io/projected/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-kube-api-access-glrng\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.671704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.671478    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/936e8f72-0cd7-4d59-abf4-9f74fba04ab2-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:13:39.864185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.864145    5784 generic.go:356] "Generic (PLEG): container finished" podID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerID="a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50" exitCode=0
Nov 03 12:13:39.864356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.864219    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"
Nov 03 12:13:39.864356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.864233    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" event={"ID":"936e8f72-0cd7-4d59-abf4-9f74fba04ab2","Type":"ContainerDied","Data":"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"}
Nov 03 12:13:39.864356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.864306    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x" event={"ID":"936e8f72-0cd7-4d59-abf4-9f74fba04ab2","Type":"ContainerDied","Data":"9cb552ea8e1ba796a0312751205e42706ee1599b21d2fface74b92b799717164"}
Nov 03 12:13:39.864356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.864328    5784 scope.go:122] "RemoveContainer" containerID="a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"
Nov 03 12:13:39.874781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.874757    5784 scope.go:122] "RemoveContainer" containerID="a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"
Nov 03 12:13:39.875058 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:13:39.875037    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50\": container with ID starting with a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50 not found: ID does not exist" containerID="a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"
Nov 03 12:13:39.875148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:39.875066    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50"} err="failed to get container status \"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50\": rpc error: code = NotFound desc = could not find container \"a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50\": container with ID starting with a77729fc10feb8165726eb0ae7b532a9f627f1abad9d5e3208ce4312bf11ae50 not found: ID does not exist"
Nov 03 12:13:40.085337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:40.085245    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:13:40.098125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:40.098085    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-8ml4x"]
Nov 03 12:13:40.236461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:40.236410    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:40.420404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:40.417172    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:41.236739 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:41.229922    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:41.463757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:41.463708    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:41.716406 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:41.716349    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" path="/var/lib/kubelet/pods/936e8f72-0cd7-4d59-abf4-9f74fba04ab2/volumes"
Nov 03 12:13:41.727522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:41.727484    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.007285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.007180    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:42.144708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.144672    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:42.145066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.145033    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.190996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.190908    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:42.477083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.465489    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:42.477083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.466288    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:13:42.498430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.491463    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.662451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.650007    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": context deadline exceeded"
Nov 03 12:13:42.662451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.651069    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:42.733461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.731475    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:42.733461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:42.732250    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:13:43.337236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:43.337203    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:43.468402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:43.466472    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.741415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:43.717482    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.741415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:43.740469    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:43.806411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:43.805328    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:13:44.059460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.058488    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:44.059460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.058558    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.242238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.242198    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:44.242532 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.242499    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.565176 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.553706    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.565176 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.557455    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 03 12:13:44.772453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.722106    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.772453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.759182    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:44.890603 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:44.890570    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 12:13:45.458402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.457596    5784 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.521402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.521150    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 12:13:45.554403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.552641    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 12:13:45.573352 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.573309    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:45.574354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.574330    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:13:45.873362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:45.873272    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 12:13:46.149403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.138895    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:46.174398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.161479    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:46.174398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.161551    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.223402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.214489    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:46.247402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.223660    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.264541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.264505    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:13:46.264848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.264801    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:13:46.362223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:46.362184    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:47.599241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:47.599205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:47.917141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:47.917101    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:49.455185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:49.455151    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:49.621442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:49.621404    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:49.785477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:49.785366    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:52.014460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:52.014426    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:52.317076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:52.316971    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:52.478167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:52.478095    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:52.656950 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:52.656901    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:52.831216 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:52.831187    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:53.387524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:53.387481    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:53.545834 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:53.545796    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:53.887012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:53.886971    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:56.472621 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:13:56.472579    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:13:57.528500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:57.528462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:59.619436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:59.619399    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:13:59.790424 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:13:59.790351    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:02.122391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:02.122349    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:06.475715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:06.474491    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.526253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:06.525482    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.685497 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:06.685452    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.748457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:06.746459    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:14:06.875323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:06.875285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:07.449630 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:07.449591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:07.602638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:07.602596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:10.209559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:10.209498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:12.114263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:12.114169    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:13.246671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:13.246638    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:16.642000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:16.641935    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:21.023745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:21.023704    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:25.372471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:25.372432    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:25.532620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:25.532580    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:25.684669 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:25.684633    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:26.474361 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:14:26.474322    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:14:29.874292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:29.874263    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:30.030574 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:30.030541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:30.189690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:30.189651    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:34.512360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:34.512324    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:38.908661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:38.908625    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:43.410341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:43.410308    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:44.038444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:44.038410    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:47.842575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:47.842541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:48.222568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:48.222525    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:48.371308 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:48.371272    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:50.800824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:50.800789    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:50.972609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:50.972561    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:52.211775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:52.211723    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:14:56.473310 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:14:56.473273    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:14:56.635289 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:14:56.635247    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:00.178891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.178853    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:00.179769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.179748    5784 cpu_manager.go:414] "containerMap: removing container" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy"
Nov 03 12:15:00.179769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.179769    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy"
Nov 03 12:15:00.179949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.179938    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="936e8f72-0cd7-4d59-abf4-9f74fba04ab2" containerName="istio-proxy"
Nov 03 12:15:00.182797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.182773    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:00.182933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.182914    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.187269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.187248    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.231297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.231257    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:15:00.266813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.266780    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:00.280348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.280313    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpscv\" (UniqueName: \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.280550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.280391    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.280550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.280539    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.280679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.280609    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4zjh\" (UniqueName: \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.281582 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.281559    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:00.382187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.382147    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4zjh\" (UniqueName: \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.382394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.382207    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpscv\" (UniqueName: \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.382394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.382252    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.382529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.382390    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.382951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.382930    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.383141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.383117    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.436963 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.436879    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpscv\" (UniqueName: \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\") pod \"maas-api-key-cleanup-29895135-8lw5r\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.437131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.437056    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4zjh\" (UniqueName: \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\") pod \"maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.500214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.500184    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:00.505325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.505306    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:00.881487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.881452    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:00.882368 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:15:00.882310    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266d7d11_6626_4528_8d4f_a29de970c114.slice/crio-d1835a3177dec84f129169d16401df3e74a136ce30120c8f6020b94dd8e25247 WatchSource:0}: Error finding container d1835a3177dec84f129169d16401df3e74a136ce30120c8f6020b94dd8e25247: Status 404 returned error can't find the container with id d1835a3177dec84f129169d16401df3e74a136ce30120c8f6020b94dd8e25247
Nov 03 12:15:00.931113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.929788    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:00.932918 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:15:00.932101    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00506d3_322c_4d57_b726_5676bd8f41a8.slice/crio-cd092f1f1ce26b46a0cfaae2287702aba18705535d95f96b9eff2447cb825a3e WatchSource:0}: Error finding container cd092f1f1ce26b46a0cfaae2287702aba18705535d95f96b9eff2447cb825a3e: Status 404 returned error can't find the container with id cd092f1f1ce26b46a0cfaae2287702aba18705535d95f96b9eff2447cb825a3e
Nov 03 12:15:00.981727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:00.981700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:01.140468 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:01.140432    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:01.280208 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:01.279894    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerStarted","Data":"cd092f1f1ce26b46a0cfaae2287702aba18705535d95f96b9eff2447cb825a3e"}
Nov 03 12:15:01.282130 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:01.282088    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerStarted","Data":"d1835a3177dec84f129169d16401df3e74a136ce30120c8f6020b94dd8e25247"}
Nov 03 12:15:02.290422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.290366    5784 generic.go:356] "Generic (PLEG): container finished" podID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerID="c2c6e7f026a8a0073ce5cd12e57169de165cc920d3d7cf491fdca510be45f96b" exitCode=60
Nov 03 12:15:02.290857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.290449    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerDied","Data":"c2c6e7f026a8a0073ce5cd12e57169de165cc920d3d7cf491fdca510be45f96b"}
Nov 03 12:15:02.290857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.290784    5784 scope.go:122] "RemoveContainer" containerID="c2c6e7f026a8a0073ce5cd12e57169de165cc920d3d7cf491fdca510be45f96b"
Nov 03 12:15:02.292194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.292170    5784 generic.go:356] "Generic (PLEG): container finished" podID="266d7d11-6626-4528-8d4f-a29de970c114" containerID="5ac548f5dce0162565859277ac93496b8525616855f03fd6875f208755acfb9b" exitCode=60
Nov 03 12:15:02.292293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.292276    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerDied","Data":"5ac548f5dce0162565859277ac93496b8525616855f03fd6875f208755acfb9b"}
Nov 03 12:15:02.292486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:02.292473    5784 scope.go:122] "RemoveContainer" containerID="5ac548f5dce0162565859277ac93496b8525616855f03fd6875f208755acfb9b"
Nov 03 12:15:03.301315 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.301277    5784 generic.go:356] "Generic (PLEG): container finished" podID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerID="b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d" exitCode=60
Nov 03 12:15:03.301836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.301359    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerDied","Data":"b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d"}
Nov 03 12:15:03.301836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.301441    5784 scope.go:122] "RemoveContainer" containerID="c2c6e7f026a8a0073ce5cd12e57169de165cc920d3d7cf491fdca510be45f96b"
Nov 03 12:15:03.301836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.301733    5784 scope.go:122] "RemoveContainer" containerID="b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d"
Nov 03 12:15:03.302112 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:03.302089    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895135-8lw5r_odh-ai-gateway-infra(a00506d3-322c-4d57-b726-5676bd8f41a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8"
Nov 03 12:15:03.303367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.303348    5784 generic.go:356] "Generic (PLEG): container finished" podID="266d7d11-6626-4528-8d4f-a29de970c114" containerID="3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e" exitCode=60
Nov 03 12:15:03.303474 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.303415    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerDied","Data":"3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e"}
Nov 03 12:15:03.303688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.303671    5784 scope.go:122] "RemoveContainer" containerID="3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e"
Nov 03 12:15:03.303943 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:03.303924    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5_odh-ai-gateway-infra(266d7d11-6626-4528-8d4f-a29de970c114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" podUID="266d7d11-6626-4528-8d4f-a29de970c114"
Nov 03 12:15:03.313246 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:03.313227    5784 scope.go:122] "RemoveContainer" containerID="5ac548f5dce0162565859277ac93496b8525616855f03fd6875f208755acfb9b"
Nov 03 12:15:04.323644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:04.323616    5784 scope.go:122] "RemoveContainer" containerID="3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e"
Nov 03 12:15:04.324115 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:04.323968    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5_odh-ai-gateway-infra(266d7d11-6626-4528-8d4f-a29de970c114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" podUID="266d7d11-6626-4528-8d4f-a29de970c114"
Nov 03 12:15:04.325699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:04.325680    5784 scope.go:122] "RemoveContainer" containerID="b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d"
Nov 03 12:15:04.325905 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:04.325888    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895135-8lw5r_odh-ai-gateway-infra(a00506d3-322c-4d57-b726-5676bd8f41a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8"
Nov 03 12:15:05.542571 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:05.542522    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:10.021993 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:10.021938    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:13.681126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:13.681089    5784 scope.go:122] "RemoveContainer" containerID="b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d"
Nov 03 12:15:13.681655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:13.681424    5784 scope.go:122] "RemoveContainer" containerID="3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e"
Nov 03 12:15:14.399708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:14.399672    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerStarted","Data":"a37d05f17917e360d2bda5418212521e655c0b22d1ccd03884f90068df2d9e87"}
Nov 03 12:15:14.401606 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:14.401575    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerStarted","Data":"1e1192cc984ed8653579659ed3b1ae4e9ffcf6dd867cd80d62f70960404a3c0d"}
Nov 03 12:15:14.504587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:14.504533    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" podStartSLOduration=14.50451861 podStartE2EDuration="14.50451861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:14.503748814 +0000 UTC m=+1420.945951134" watchObservedRunningTime="2026-11-03 12:15:14.50451861 +0000 UTC m=+1420.946720915"
Nov 03 12:15:14.611165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:14.611098    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" podStartSLOduration=14.611081584 podStartE2EDuration="14.611081584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:15:14.60750531 +0000 UTC m=+1421.049707648" watchObservedRunningTime="2026-11-03 12:15:14.611081584 +0000 UTC m=+1421.053283890"
Nov 03 12:15:14.931038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:14.931003    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:15.410584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.410550    5784 generic.go:356] "Generic (PLEG): container finished" podID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerID="a37d05f17917e360d2bda5418212521e655c0b22d1ccd03884f90068df2d9e87" exitCode=60
Nov 03 12:15:15.410742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.410621    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerDied","Data":"a37d05f17917e360d2bda5418212521e655c0b22d1ccd03884f90068df2d9e87"}
Nov 03 12:15:15.410742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.410675    5784 scope.go:122] "RemoveContainer" containerID="b09dab96b14c0bf2537f35b9d455e30fd93c7ae5ec53cc43047814a9d4498a2d"
Nov 03 12:15:15.411051 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.411030    5784 scope.go:122] "RemoveContainer" containerID="a37d05f17917e360d2bda5418212521e655c0b22d1ccd03884f90068df2d9e87"
Nov 03 12:15:15.411353 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:15.411334    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895135-8lw5r_odh-ai-gateway-infra(a00506d3-322c-4d57-b726-5676bd8f41a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8"
Nov 03 12:15:15.412613 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.412590    5784 generic.go:356] "Generic (PLEG): container finished" podID="266d7d11-6626-4528-8d4f-a29de970c114" containerID="1e1192cc984ed8653579659ed3b1ae4e9ffcf6dd867cd80d62f70960404a3c0d" exitCode=60
Nov 03 12:15:15.412712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.412650    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerDied","Data":"1e1192cc984ed8653579659ed3b1ae4e9ffcf6dd867cd80d62f70960404a3c0d"}
Nov 03 12:15:15.413010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.412992    5784 scope.go:122] "RemoveContainer" containerID="1e1192cc984ed8653579659ed3b1ae4e9ffcf6dd867cd80d62f70960404a3c0d"
Nov 03 12:15:15.413239 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:15.413223    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5_odh-ai-gateway-infra(266d7d11-6626-4528-8d4f-a29de970c114)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" podUID="266d7d11-6626-4528-8d4f-a29de970c114"
Nov 03 12:15:15.424564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.424540    5784 scope.go:122] "RemoveContainer" containerID="3b31116fce8c0b4f0867efd01124b5e2973d5796152a483cbf05f86ecc112d0e"
Nov 03 12:15:15.515541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.515505    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:15.618731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:15.618686    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:16.593804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.593776    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:16.602946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.602912    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\") pod \"266d7d11-6626-4528-8d4f-a29de970c114\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") "
Nov 03 12:15:16.603083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.603064    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\" (UniqueName: \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\") pod \"266d7d11-6626-4528-8d4f-a29de970c114\" (UID: \"266d7d11-6626-4528-8d4f-a29de970c114\") "
Nov 03 12:15:16.604244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.604218    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca" pod "266d7d11-6626-4528-8d4f-a29de970c114" (UID: "266d7d11-6626-4528-8d4f-a29de970c114"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:16.606321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.606283    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh" pod "266d7d11-6626-4528-8d4f-a29de970c114" (UID: "266d7d11-6626-4528-8d4f-a29de970c114"). InnerVolumeSpecName "kube-api-access-t4zjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:16.669554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.669527    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:16.704274 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704204    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\") pod \"a00506d3-322c-4d57-b726-5676bd8f41a8\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") "
Nov 03 12:15:16.704424 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704308    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\" (UniqueName: \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\") pod \"a00506d3-322c-4d57-b726-5676bd8f41a8\" (UID: \"a00506d3-322c-4d57-b726-5676bd8f41a8\") "
Nov 03 12:15:16.704561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704539    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca" pod "a00506d3-322c-4d57-b726-5676bd8f41a8" (UID: "a00506d3-322c-4d57-b726-5676bd8f41a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:15:16.704755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704732    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/266d7d11-6626-4528-8d4f-a29de970c114-maas-api-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:16.704755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704750    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a00506d3-322c-4d57-b726-5676bd8f41a8-maas-api-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:16.704755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.704760    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4zjh\" (UniqueName: \"kubernetes.io/projected/266d7d11-6626-4528-8d4f-a29de970c114-kube-api-access-t4zjh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:16.706393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.706362    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv" pod "a00506d3-322c-4d57-b726-5676bd8f41a8" (UID: "a00506d3-322c-4d57-b726-5676bd8f41a8"). InnerVolumeSpecName "kube-api-access-jpscv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:15:16.806435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:16.806368    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpscv\" (UniqueName: \"kubernetes.io/projected/a00506d3-322c-4d57-b726-5676bd8f41a8-kube-api-access-jpscv\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:15:17.434262 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.434228    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"
Nov 03 12:15:17.434262 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.434248    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5" event={"ID":"266d7d11-6626-4528-8d4f-a29de970c114","Type":"ContainerDied","Data":"d1835a3177dec84f129169d16401df3e74a136ce30120c8f6020b94dd8e25247"}
Nov 03 12:15:17.434518 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.434313    5784 scope.go:122] "RemoveContainer" containerID="1e1192cc984ed8653579659ed3b1ae4e9ffcf6dd867cd80d62f70960404a3c0d"
Nov 03 12:15:17.435957 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.435933    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"
Nov 03 12:15:17.436073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.435931    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r" event={"ID":"a00506d3-322c-4d57-b726-5676bd8f41a8","Type":"ContainerDied","Data":"cd092f1f1ce26b46a0cfaae2287702aba18705535d95f96b9eff2447cb825a3e"}
Nov 03 12:15:17.446959 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.446933    5784 scope.go:122] "RemoveContainer" containerID="a37d05f17917e360d2bda5418212521e655c0b22d1ccd03884f90068df2d9e87"
Nov 03 12:15:17.587369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.587336    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:17.632655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.632624    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-ac6071-29895135-d2tg5"]
Nov 03 12:15:17.684157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.684129    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="266d7d11-6626-4528-8d4f-a29de970c114" path="/var/lib/kubelet/pods/266d7d11-6626-4528-8d4f-a29de970c114/volumes"
Nov 03 12:15:17.688771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.688249    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:17.692009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:17.691979    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895135-8lw5r"]
Nov 03 12:15:18.048625 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:18.048539    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:18.796129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:18.796086    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:19.583510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:19.583474    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:19.684606 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:19.684568    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" path="/var/lib/kubelet/pods/a00506d3-322c-4d57-b726-5676bd8f41a8/volumes"
Nov 03 12:15:19.773407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:19.772883    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:19.957773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:19.957715    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:20.892249 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:20.892205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:21.623500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:21.623462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:22.219443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:22.219413    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:22.605430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:22.605310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:23.667854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:23.667820    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:24.333583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:24.333547    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:24.568481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:24.568449    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:24.749894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:24.749859    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:26.476545 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:26.476497    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:15:27.412628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:27.412583    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:27.570575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:27.570538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:27.741316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:27.741220    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:29.501917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:29.501884    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:32.065359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:32.065321    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:32.230082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:32.228826    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:34.037925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:34.037901    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:38.420541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:38.420504    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:38.617288 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:38.617252    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:39.045120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:39.045078    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:39.219657 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:39.219630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:43.088934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:43.088849    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:43.265190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:43.265153    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:46.114430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:46.114387    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:46.273973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:46.273934    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:47.925359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:47.925326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:48.556428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:48.556387    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:52.510806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:52.510771    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:15:56.473942 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:15:56.473906    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:15:56.895636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:15:56.895601    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:01.365199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:01.365162    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:05.454537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.454499    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:16:05.456355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456326    5784 cpu_manager.go:414] "containerMap: removing container" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.456561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456545    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.456685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456673    5784 cpu_manager.go:414] "containerMap: removing container" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.456776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456764    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.456912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456900    5784 cpu_manager.go:414] "containerMap: removing container" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.456992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.456982    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.457127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457117    5784 cpu_manager.go:414] "containerMap: removing container" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.457214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457198    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.457584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457567    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.457697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457686    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:05.457776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457768    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.457867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.457858    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:05.463423 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.463403    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.508747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.508716    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:16:05.548049 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.548009    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.548198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.548068    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.548198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.548147    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.548269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.548204    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dq87\" (UniqueName: \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.550099 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.550078    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-913a17-serving-cert\""
Nov 03 12:16:05.649673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.649638    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.649882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.649698    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.649882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.649727    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.649882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.649782    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dq87\" (UniqueName: \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.649882 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:05.649823    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-serving-cert: secret "maas-api-e2e-api-b-913a17-serving-cert" not found
Nov 03 12:16:05.650115 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:05.649926    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls podName:72b5cdfc-2f58-44a8-a2c0-12e686fb563e nodeName:}" failed. No retries permitted until 2026-11-03 12:16:06.149903 +0000 UTC m=+1472.592105297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls") pod "maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" (UID: "72b5cdfc-2f58-44a8-a2c0-12e686fb563e") : secret "maas-api-e2e-api-b-913a17-serving-cert" not found
Nov 03 12:16:05.650510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.650486    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.652550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.652525    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.653295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.653270    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:05.705681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.705595    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dq87\" (UniqueName: \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:05.803173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.803142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:05.960472 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:05.960394    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:06.156645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:06.156608    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:06.159035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:06.159010    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") pod \"maas-api-e2e-api-b-913a17-688dbf6cd6-284q5\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:06.378841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:06.378810    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:06.756722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:06.756694    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:16:06.758620 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:16:06.758585    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b5cdfc_2f58_44a8_a2c0_12e686fb563e.slice/crio-2005536f5507666a293814bb8c37239a5286bda31a458f80f27c489c32bea8c2 WatchSource:0}: Error finding container 2005536f5507666a293814bb8c37239a5286bda31a458f80f27c489c32bea8c2: Status 404 returned error can't find the container with id 2005536f5507666a293814bb8c37239a5286bda31a458f80f27c489c32bea8c2
Nov 03 12:16:06.933419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:06.933363    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" event={"ID":"72b5cdfc-2f58-44a8-a2c0-12e686fb563e","Type":"ContainerStarted","Data":"2005536f5507666a293814bb8c37239a5286bda31a458f80f27c489c32bea8c2"}
Nov 03 12:16:07.945708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:07.945620    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" event={"ID":"72b5cdfc-2f58-44a8-a2c0-12e686fb563e","Type":"ContainerStarted","Data":"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"}
Nov 03 12:16:07.946125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:07.945748    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:08.056726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:08.056670    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" podStartSLOduration=2.18730115 podStartE2EDuration="3.056646746s" totalImagesPullingTime="869.345596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:08.052555251 +0000 UTC m=+1474.494757556" watchObservedRunningTime="2026-11-03 12:16:08.056646746 +0000 UTC m=+1474.498849052"
Nov 03 12:16:10.312728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:10.312690    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:13.960770 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:13.960738    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:16:14.458391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.458340    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:16:14.459193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459165    5784 cpu_manager.go:414] "containerMap: removing container" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:14.459193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459187    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:14.459330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459215    5784 cpu_manager.go:414] "containerMap: removing container" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:14.459330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459220    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:14.459330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459320    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="266d7d11-6626-4528-8d4f-a29de970c114" containerName="cleanup"
Nov 03 12:16:14.459330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.459327    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00506d3-322c-4d57-b726-5676bd8f41a8" containerName="cleanup"
Nov 03 12:16:14.462820 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.462801    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.512373 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.512329    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-931523-serving-cert\""
Nov 03 12:16:14.517418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.517364    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:16:14.563280 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.563243    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.563485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.563306    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.563584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.563529    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snz5s\" (UniqueName: \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.563640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.563593    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.660912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.660876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:14.664419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.664387    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snz5s\" (UniqueName: \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.664555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.664459    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.664644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.664617    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.664708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.664672    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.664843 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:14.664820    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-931523-serving-cert: secret "maas-api-e2e-api-b-931523-serving-cert" not found
Nov 03 12:16:14.664921 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:14.664904    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls podName:b2bd1652-04f9-4b0b-8da8-4a9f537ef40e nodeName:}" failed. No retries permitted until 2026-11-03 12:16:15.164885894 +0000 UTC m=+1481.607088193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls") pod "maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" (UID: "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e") : secret "maas-api-e2e-api-b-931523-serving-cert" not found
Nov 03 12:16:14.665299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.665277    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.667249 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.667224    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.729292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.729183    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-dfd3db"]
Nov 03 12:16:14.734761 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.734685    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snz5s\" (UniqueName: \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:14.741040 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.739587    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:14.791414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.787999    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 12:16:14.814035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.813448    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-dfd3db"]
Nov 03 12:16:14.814227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.814101    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:14.867866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.867826    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk64l\" (UniqueName: \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\") pod \"test-curl-7593-dfd3db\" (UID: \"d897892d-b19f-4ca2-bf19-1875f1e4e7fe\") " pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:14.969888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.969843    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk64l\" (UniqueName: \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\") pod \"test-curl-7593-dfd3db\" (UID: \"d897892d-b19f-4ca2-bf19-1875f1e4e7fe\") " pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:14.973786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:14.973754    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:15.022148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.022075    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk64l\" (UniqueName: \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\") pod \"test-curl-7593-dfd3db\" (UID: \"d897892d-b19f-4ca2-bf19-1875f1e4e7fe\") " pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:15.060019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.059989    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:15.171963 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.171926    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:15.174247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.174224    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") pod \"maas-api-e2e-api-b-931523-76b6f799b9-b5tt4\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:15.378967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.378937    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:15.424522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.423477    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-dfd3db"]
Nov 03 12:16:15.425543 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:16:15.425514    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd897892d_b19f_4ca2_bf19_1875f1e4e7fe.slice/crio-3c9e25456294ff65aa54e808966ee19605e8cb97df798558c0919c7b7b21a967 WatchSource:0}: Error finding container 3c9e25456294ff65aa54e808966ee19605e8cb97df798558c0919c7b7b21a967: Status 404 returned error can't find the container with id 3c9e25456294ff65aa54e808966ee19605e8cb97df798558c0919c7b7b21a967
Nov 03 12:16:15.796706 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:16:15.796673    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bd1652_04f9_4b0b_8da8_4a9f537ef40e.slice/crio-99452bad7863f3ced06514f41af16db707c984f832d950ba647cde0531fb26ca WatchSource:0}: Error finding container 99452bad7863f3ced06514f41af16db707c984f832d950ba647cde0531fb26ca: Status 404 returned error can't find the container with id 99452bad7863f3ced06514f41af16db707c984f832d950ba647cde0531fb26ca
Nov 03 12:16:15.796706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:15.796690    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:16:16.005087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:16.005054    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-dfd3db" event={"ID":"d897892d-b19f-4ca2-bf19-1875f1e4e7fe","Type":"ContainerStarted","Data":"3c9e25456294ff65aa54e808966ee19605e8cb97df798558c0919c7b7b21a967"}
Nov 03 12:16:16.006333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:16.006301    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" event={"ID":"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e","Type":"ContainerStarted","Data":"99452bad7863f3ced06514f41af16db707c984f832d950ba647cde0531fb26ca"}
Nov 03 12:16:17.017983 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:17.017898    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" event={"ID":"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e","Type":"ContainerStarted","Data":"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"}
Nov 03 12:16:17.018543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:17.018124    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:17.124713 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:17.124637    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" podStartSLOduration=2.169350088 podStartE2EDuration="3.124618623s" totalImagesPullingTime="955.268535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:17.123123052 +0000 UTC m=+1483.565325388" watchObservedRunningTime="2026-11-03 12:16:17.124618623 +0000 UTC m=+1483.566820930"
Nov 03 12:16:19.033779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:19.033740    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-dfd3db" event={"ID":"d897892d-b19f-4ca2-bf19-1875f1e4e7fe","Type":"ContainerStarted","Data":"bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d"}
Nov 03 12:16:19.139223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:19.139172    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-dfd3db" podStartSLOduration=2.117852334 podStartE2EDuration="5.139158731s" totalImagesPullingTime="3.021306397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:19.137896285 +0000 UTC m=+1485.580098607" watchObservedRunningTime="2026-11-03 12:16:19.139158731 +0000 UTC m=+1485.581361036"
Nov 03 12:16:19.304652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:19.304551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:20.571337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:20.571290    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-dfd3db"]
Nov 03 12:16:20.575452 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:20.575423    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-dfd3db"]
Nov 03 12:16:21.047343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:21.047294    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-dfd3db" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" containerName="test-curl-7593-dfd3db" containerID="cri-o://bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d" gracePeriod=2
Nov 03 12:16:21.094581 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:21.094549    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-dfd3db\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:23.031503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.031470    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:16:23.333026 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.333000    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:23.370942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.370908    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\" (UniqueName: \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\") pod \"d897892d-b19f-4ca2-bf19-1875f1e4e7fe\" (UID: \"d897892d-b19f-4ca2-bf19-1875f1e4e7fe\") "
Nov 03 12:16:23.372997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.372968    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l" pod "d897892d-b19f-4ca2-bf19-1875f1e4e7fe" (UID: "d897892d-b19f-4ca2-bf19-1875f1e4e7fe"). InnerVolumeSpecName "kube-api-access-rk64l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:16:23.473307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.473270    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk64l\" (UniqueName: \"kubernetes.io/projected/d897892d-b19f-4ca2-bf19-1875f1e4e7fe-kube-api-access-rk64l\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:16:23.684763 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:23.684729    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" path="/var/lib/kubelet/pods/d897892d-b19f-4ca2-bf19-1875f1e4e7fe/volumes"
Nov 03 12:16:24.052078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.051992    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:24.081014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.080496    5784 generic.go:356] "Generic (PLEG): container finished" podID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" containerID="bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d" exitCode=137
Nov 03 12:16:24.081843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.081820    5784 scope.go:122] "RemoveContainer" containerID="bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d"
Nov 03 12:16:24.083628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.083229    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-dfd3db"
Nov 03 12:16:24.094445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.094422    5784 scope.go:122] "RemoveContainer" containerID="bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d"
Nov 03 12:16:24.094788 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:24.094769    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d\": container with ID starting with bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d not found: ID does not exist" containerID="bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d"
Nov 03 12:16:24.094867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:24.094800    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d"} err="failed to get container status \"bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d\": rpc error: code = NotFound desc = could not find container \"bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d\": container with ID starting with bac9be60a1c50b2f7385df614541e3fed7db5d7ec9cb11467522e9de2eae543d not found: ID does not exist"
Nov 03 12:16:26.471160 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:26.471126    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:16:29.024154 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:29.024110    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:33.176653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.176620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:33.336067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.335802    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:33.583790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.583569    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:16:33.584389 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.584347    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" containerName="test-curl-7593-dfd3db"
Nov 03 12:16:33.584389 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.584372    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" containerName="test-curl-7593-dfd3db"
Nov 03 12:16:33.584605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.584589    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d897892d-b19f-4ca2-bf19-1875f1e4e7fe" containerName="test-curl-7593-dfd3db"
Nov 03 12:16:33.588835 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.588813    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.683236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.683199    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f6c034-serving-cert\""
Nov 03 12:16:33.693657 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.693627    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:16:33.717476 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.717443    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.717649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.717501    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.717649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.717602    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.717914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.717887    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59d2\" (UniqueName: \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.794049 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.793993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:33.835450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.834482    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.835450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.834772    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59d2\" (UniqueName: \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.835450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.834836    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.835450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.834878    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.835765 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.835735    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.839396 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:33.835905    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-serving-cert: secret "maas-api-e2e-api-a-f6c034-serving-cert" not found
Nov 03 12:16:33.839396 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:33.835978    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls podName:debbe158-c45b-4d7f-a6d7-00d029742d85 nodeName:}" failed. No retries permitted until 2026-11-03 12:16:34.335958589 +0000 UTC m=+1500.778160876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls") pod "maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" (UID: "debbe158-c45b-4d7f-a6d7-00d029742d85") : secret "maas-api-e2e-api-a-f6c034-serving-cert" not found
Nov 03 12:16:33.868747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.867128    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.891139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.891101    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59d2\" (UniqueName: \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:33.932697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.932664    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:16:33.940146 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.940107    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:16:33.955594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:33.955563    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:34.161153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:34.161119    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:34.346815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:34.346782    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:34.351188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:34.351151    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") pod \"maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:34.503152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:34.503078    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:34.907316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:34.907284    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:16:34.908244 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:16:34.908208    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddebbe158_c45b_4d7f_a6d7_00d029742d85.slice/crio-347fc61ff91a2577cc755b766c9f9c9540d9d4659fecf7ca67526528eeeb31ec WatchSource:0}: Error finding container 347fc61ff91a2577cc755b766c9f9c9540d9d4659fecf7ca67526528eeeb31ec: Status 404 returned error can't find the container with id 347fc61ff91a2577cc755b766c9f9c9540d9d4659fecf7ca67526528eeeb31ec
Nov 03 12:16:35.163727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:35.163639    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" event={"ID":"debbe158-c45b-4d7f-a6d7-00d029742d85","Type":"ContainerStarted","Data":"347fc61ff91a2577cc755b766c9f9c9540d9d4659fecf7ca67526528eeeb31ec"}
Nov 03 12:16:36.172041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:36.172006    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" event={"ID":"debbe158-c45b-4d7f-a6d7-00d029742d85","Type":"ContainerStarted","Data":"9a2047209f3e0dc35764839399b7b33b44e23bd073b6b7a4ac214f0f88581cc4"}
Nov 03 12:16:36.172464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:36.172068    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:36.280741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:36.280692    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" podStartSLOduration=2.406566253 podStartE2EDuration="3.280671408s" totalImagesPullingTime="874.105155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:36.278096832 +0000 UTC m=+1502.720299158" watchObservedRunningTime="2026-11-03 12:16:36.280671408 +0000 UTC m=+1502.722873713"
Nov 03 12:16:38.802797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:38.802763    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:42.184509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:42.184478    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:16:43.355544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:43.355512    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:47.682255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:47.682216    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:48.886699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:48.886666    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:50.574751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.574707    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:16:50.579034 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.579011    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.627574 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.627544    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:16:50.664085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664050    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz67b\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664245 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664155    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664245 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664199    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664245 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664240    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664285    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664312    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664474 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664429    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664488    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.664559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.664512    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.678670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.678642    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:16:50.766192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766149    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz67b\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766214    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766249    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766311    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766338    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766362    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766721 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766444    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766721 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766551    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766721 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766592    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766747    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.766933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.766907    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.767197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.767139    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.767197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.767140    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.767365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.767214    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.768811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.768790    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.769144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.769127    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.819226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.819197    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.819337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.819310    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz67b\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:50.896888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:50.896857    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:51.315529 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:16:51.315489    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a232601_351d_41f5_9be2_86b807416807.slice/crio-5f7d03c6b732b418f74027950e5940c9e980118095190f87a93fb9b2dbbf8efd WatchSource:0}: Error finding container 5f7d03c6b732b418f74027950e5940c9e980118095190f87a93fb9b2dbbf8efd: Status 404 returned error can't find the container with id 5f7d03c6b732b418f74027950e5940c9e980118095190f87a93fb9b2dbbf8efd
Nov 03 12:16:51.315994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:51.315971    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:16:51.318694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:51.318648    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:16:51.318800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:51.318736    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:16:51.318800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:51.318776    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:16:52.117848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.117816    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:52.291074 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.291035    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" event={"ID":"2a232601-351d-41f5-9be2-86b807416807","Type":"ContainerStarted","Data":"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"}
Nov 03 12:16:52.291254 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.291094    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" event={"ID":"2a232601-351d-41f5-9be2-86b807416807","Type":"ContainerStarted","Data":"5f7d03c6b732b418f74027950e5940c9e980118095190f87a93fb9b2dbbf8efd"}
Nov 03 12:16:52.496186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.496130    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podStartSLOduration=2.496112818 podStartE2EDuration="2.496112818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:16:52.450152757 +0000 UTC m=+1518.892355062" watchObservedRunningTime="2026-11-03 12:16:52.496112818 +0000 UTC m=+1518.938315123"
Nov 03 12:16:52.897771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.897736    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:52.899478 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.899455    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:52.899552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:52.899504    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:53.897903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:53.897854    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:53.898424 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:53.897916    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:54.897372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:54.897335    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:54.897638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:54.897404    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:55.397974 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:55.397939    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:55.897819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:55.897782    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:55.897999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:55.897852    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:56.476741 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:16:56.476696    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:16:56.586538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:56.586504    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:16:56.897754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:56.897722    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:56.897928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:56.897774    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:57.898207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:57.898174    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 03 12:16:57.898623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:57.898234    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 03 12:16:59.290052 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:59.290017    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:59.290623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:59.290593    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:16:59.291596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:16:59.291574    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:17:01.143017 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:01.142976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:05.716325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:05.716283    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:06.259342 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:06.259308    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:08.597356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:08.597323    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:10.502902 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:10.502862    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:14.076545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:14.076454    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:14.496184 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:14.496157    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:14.656632 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:14.656600    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:14.819630 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:14.819545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:15.410680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:15.410653    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:16.863354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:16.863311    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:19.944642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:19.944564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:20.575044 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:20.575005    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:22.641215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:22.641119    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:22.804694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:22.804658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:23.942369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:23.942335    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:24.846955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:24.846901    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:25.019298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:25.019265    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:25.166544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:25.166508    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:26.473150 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:17:26.473112    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:17:28.247801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:28.247754    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:29.653134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:29.653052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:30.000227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:30.000127    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:30.158613 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:30.158581    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:30.336815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:30.336700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:30.502706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:30.502640    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:31.058158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:31.058124    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:31.631871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:31.631831    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:31.796402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:31.796025    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:33.864536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:33.864492    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:34.583495 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:34.583450    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:34.741337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:34.741297    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:35.148549 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:35.148429    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:36.374325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:36.374294    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:36.646225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:36.646191    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:36.814344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:36.814295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:36.971159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:36.971078    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:39.476220 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:39.476188    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:39.643894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:39.643844    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:41.605409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:41.605319    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:42.723580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:42.723545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:42.885235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:42.885204    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:44.118854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:44.118823    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:44.276844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:44.276809    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:44.616242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:44.616205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:45.394568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:45.394526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:45.668264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:45.668178    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:46.129851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:46.129817    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:46.547226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:46.547115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:46.920252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:46.920215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:49.186239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:49.186210    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:49.333071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:49.333036    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:50.478971 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.478930    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:17:50.479421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.479171    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerName="maas-api" containerID="cri-o://f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee" gracePeriod=30
Nov 03 12:17:50.597749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.597716    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:50.820729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.820692    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:17:50.957344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.957293    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\") pod \"81cfc45f-842f-4d4c-b316-4a22179cf697\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") "
Nov 03 12:17:50.957561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.957433    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") pod \"81cfc45f-842f-4d4c-b316-4a22179cf697\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") "
Nov 03 12:17:50.957628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.957575    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\") pod \"81cfc45f-842f-4d4c-b316-4a22179cf697\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") "
Nov 03 12:17:50.957689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.957635    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\" (UniqueName: \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\") pod \"81cfc45f-842f-4d4c-b316-4a22179cf697\" (UID: \"81cfc45f-842f-4d4c-b316-4a22179cf697\") "
Nov 03 12:17:50.962793 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.962756    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8" pod "81cfc45f-842f-4d4c-b316-4a22179cf697" (UID: "81cfc45f-842f-4d4c-b316-4a22179cf697"). InnerVolumeSpecName "kube-api-access-n6bz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:17:50.964432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.964391    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca" pod "81cfc45f-842f-4d4c-b316-4a22179cf697" (UID: "81cfc45f-842f-4d4c-b316-4a22179cf697"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:17:50.965655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.965629    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls" pod "81cfc45f-842f-4d4c-b316-4a22179cf697" (UID: "81cfc45f-842f-4d4c-b316-4a22179cf697"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:50.967689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.967662    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs" pod "81cfc45f-842f-4d4c-b316-4a22179cf697" (UID: "81cfc45f-842f-4d4c-b316-4a22179cf697"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:17:50.977671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.977625    5784 generic.go:356] "Generic (PLEG): container finished" podID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerID="f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee" exitCode=0
Nov 03 12:17:50.978273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.977841    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" event={"ID":"81cfc45f-842f-4d4c-b316-4a22179cf697","Type":"ContainerDied","Data":"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"}
Nov 03 12:17:50.978273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.978017    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" event={"ID":"81cfc45f-842f-4d4c-b316-4a22179cf697","Type":"ContainerDied","Data":"e97ee3a527d14f932d8a7ff972b5f3282ef5baedcad65a882642972664839474"}
Nov 03 12:17:50.978273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.978058    5784 scope.go:122] "RemoveContainer" containerID="f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"
Nov 03 12:17:50.978273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.977873    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"
Nov 03 12:17:50.999405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.998555    5784 scope.go:122] "RemoveContainer" containerID="f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"
Nov 03 12:17:50.999405 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:17:50.998891    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee\": container with ID starting with f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee not found: ID does not exist" containerID="f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"
Nov 03 12:17:50.999405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:50.998922    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee"} err="failed to get container status \"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee\": rpc error: code = NotFound desc = could not find container \"f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee\": container with ID starting with f4809d7680c4d47b7d90913c35c7567117a2e3ee4b1855464d3d71bbea9fe2ee not found: ID does not exist"
Nov 03 12:17:51.060546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.060468    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81cfc45f-842f-4d4c-b316-4a22179cf697-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.060546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.060493    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.060546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.060502    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81cfc45f-842f-4d4c-b316-4a22179cf697-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.060546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.060511    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6bz8\" (UniqueName: \"kubernetes.io/projected/81cfc45f-842f-4d4c-b316-4a22179cf697-kube-api-access-n6bz8\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:17:51.135066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.135025    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:17:51.138909 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.138879    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf"]
Nov 03 12:17:51.476172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.475974    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:51.684696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:51.684659    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" path="/var/lib/kubelet/pods/81cfc45f-842f-4d4c-b316-4a22179cf697/volumes"
Nov 03 12:17:52.174364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:52.174331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:52.425165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:52.425082    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:52.576992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:52.576955    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:52.734043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:52.733958    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:53.765268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:53.765200    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:53.790113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:53.790076    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69fd8043-64675459d-4jqbf" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.84:8443/health\": context deadline exceeded"
Nov 03 12:17:54.424940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:54.424908    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:54.583329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:54.583280    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:56.474300 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:17:56.474267    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:17:56.686888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:56.686854    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:56.879522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:56.879495    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:57.191725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:57.191635    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:17:59.182175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:17:59.182136    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:00.276547 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:00.276512    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:00.512587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:00.512554    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:02.010244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:02.010208    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:02.536431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:02.532649    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:02.726931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:02.726789    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:07.066063 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:07.066033    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:11.371276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:11.371247    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:11.533985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:11.533928    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:11.684020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:11.683987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:15.940863 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:15.940828    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:20.067562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.067519    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:18:20.068173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.068157    5784 cpu_manager.go:414] "containerMap: removing container" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerName="maas-api"
Nov 03 12:18:20.068224 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.068175    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerName="maas-api"
Nov 03 12:18:20.068289 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.068279    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cfc45f-842f-4d4c-b316-4a22179cf697" containerName="maas-api"
Nov 03 12:18:20.072034 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.072013    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.121223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.121188    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-89a88d-serving-cert\""
Nov 03 12:18:20.174002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.173970    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:18:20.194706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.193398    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.194706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.193597    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.194706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.193671    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.194706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.193704    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshsn\" (UniqueName: \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.287994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.287959    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:20.295335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.295306    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.295515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.295478    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.295585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.295518    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshsn\" (UniqueName: \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.295711 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.295697    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.296240 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:20.295843    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-serving-cert: secret "maas-api-e2e-api-b-89a88d-serving-cert" not found
Nov 03 12:18:20.296240 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:20.295932    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls podName:23b90192-c3e5-4453-b5a9-81b1d9467e5a nodeName:}" failed. No retries permitted until 2026-11-03 12:18:20.795909234 +0000 UTC m=+1607.238111537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls") pod "maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" (UID: "23b90192-c3e5-4453-b5a9-81b1d9467e5a") : secret "maas-api-e2e-api-b-89a88d-serving-cert" not found
Nov 03 12:18:20.296355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.296283    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.298483 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.298438    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.353015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.352935    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshsn\" (UniqueName: \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.449201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.449166    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:20.599327 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.599294    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:20.802813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.802683    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.805303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.805278    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") pod \"maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:20.987367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:20.987333    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:21.361708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:21.361678    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:18:21.362439 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:18:21.362395    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b90192_c3e5_4453_b5a9_81b1d9467e5a.slice/crio-47675088e8d82e22922be97f609f0b0eedc238f66ab61197268934205315ea7d WatchSource:0}: Error finding container 47675088e8d82e22922be97f609f0b0eedc238f66ab61197268934205315ea7d: Status 404 returned error can't find the container with id 47675088e8d82e22922be97f609f0b0eedc238f66ab61197268934205315ea7d
Nov 03 12:18:22.367447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:22.367410    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" event={"ID":"23b90192-c3e5-4453-b5a9-81b1d9467e5a","Type":"ContainerStarted","Data":"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"}
Nov 03 12:18:22.367855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:22.367473    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" event={"ID":"23b90192-c3e5-4453-b5a9-81b1d9467e5a","Type":"ContainerStarted","Data":"47675088e8d82e22922be97f609f0b0eedc238f66ab61197268934205315ea7d"}
Nov 03 12:18:22.367855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:22.367619    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:22.477055 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:22.477003    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" podStartSLOduration=1.574942544 podStartE2EDuration="2.476988715s" totalImagesPullingTime="902.046171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:18:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:18:22.474790247 +0000 UTC m=+1608.916992551" watchObservedRunningTime="2026-11-03 12:18:22.476988715 +0000 UTC m=+1608.919191010"
Nov 03 12:18:24.833089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:24.833052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:26.473815 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:26.473775    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:18:28.381779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:28.381749    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:18:29.345711 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:29.345677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:33.008082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.008048    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:18:33.009041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.009012    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" podUID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" containerName="maas-api" containerID="cri-o://c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8" gracePeriod=30
Nov 03 12:18:33.136692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.136655    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:33.400255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.400086    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:18:33.463400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.462434    5784 generic.go:356] "Generic (PLEG): container finished" podID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" containerID="c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8" exitCode=0
Nov 03 12:18:33.463400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.462672    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" event={"ID":"72b5cdfc-2f58-44a8-a2c0-12e686fb563e","Type":"ContainerDied","Data":"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"}
Nov 03 12:18:33.463400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.462723    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5" event={"ID":"72b5cdfc-2f58-44a8-a2c0-12e686fb563e","Type":"ContainerDied","Data":"2005536f5507666a293814bb8c37239a5286bda31a458f80f27c489c32bea8c2"}
Nov 03 12:18:33.463400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.462756    5784 scope.go:122] "RemoveContainer" containerID="c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"
Nov 03 12:18:33.463400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.462918    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"
Nov 03 12:18:33.470572 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.470214    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\") pod \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") "
Nov 03 12:18:33.478402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.476828    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") pod \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") "
Nov 03 12:18:33.478402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.477423    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\") pod \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") "
Nov 03 12:18:33.478402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.477764    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca" pod "72b5cdfc-2f58-44a8-a2c0-12e686fb563e" (UID: "72b5cdfc-2f58-44a8-a2c0-12e686fb563e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:33.478740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.477991    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\" (UniqueName: \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\") pod \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\" (UID: \"72b5cdfc-2f58-44a8-a2c0-12e686fb563e\") "
Nov 03 12:18:33.483196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.479587    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs" pod "72b5cdfc-2f58-44a8-a2c0-12e686fb563e" (UID: "72b5cdfc-2f58-44a8-a2c0-12e686fb563e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:18:33.495402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.484495    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:33.495402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.484539    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:33.495402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.495301    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87" pod "72b5cdfc-2f58-44a8-a2c0-12e686fb563e" (UID: "72b5cdfc-2f58-44a8-a2c0-12e686fb563e"). InnerVolumeSpecName "kube-api-access-8dq87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:33.505540 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.505486    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls" pod "72b5cdfc-2f58-44a8-a2c0-12e686fb563e" (UID: "72b5cdfc-2f58-44a8-a2c0-12e686fb563e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:18:33.529762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.529665    5784 scope.go:122] "RemoveContainer" containerID="c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"
Nov 03 12:18:33.530264 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:33.530179    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8\": container with ID starting with c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8 not found: ID does not exist" containerID="c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"
Nov 03 12:18:33.530264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.530214    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8"} err="failed to get container status \"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8\": rpc error: code = NotFound desc = could not find container \"c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8\": container with ID starting with c673b56c72fd2e4b5cbd1f7757404cca02184fe96453195a66f55ffb529221c8 not found: ID does not exist"
Nov 03 12:18:33.585396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.585354    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:33.585396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.585398    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dq87\" (UniqueName: \"kubernetes.io/projected/72b5cdfc-2f58-44a8-a2c0-12e686fb563e-kube-api-access-8dq87\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:33.870529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.870493    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:18:33.874163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:33.874138    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-913a17-688dbf6cd6-284q5"]
Nov 03 12:18:34.658486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:34.658436    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:34.824016 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:34.823540    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:34.980777 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:34.980657    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:35.683462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:35.683430    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" path="/var/lib/kubelet/pods/72b5cdfc-2f58-44a8-a2c0-12e686fb563e/volumes"
Nov 03 12:18:36.187162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:36.187119    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:18:36.187393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:36.187338    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" containerID="cri-o://38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54" gracePeriod=30
Nov 03 12:18:38.619854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:38.619816    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:18:38.620335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:38.620074    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" podUID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" containerName="maas-api" containerID="cri-o://6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c" gracePeriod=30
Nov 03 12:18:38.735719 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:38.735613    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:38.981492 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:38.981467    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:18:39.092399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.088762    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\" (UniqueName: \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\") pod \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") "
Nov 03 12:18:39.092399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.088818    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\") pod \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") "
Nov 03 12:18:39.092399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.089012    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\") pod \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") "
Nov 03 12:18:39.092399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.089088    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") pod \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\" (UID: \"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e\") "
Nov 03 12:18:39.092399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.090738    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca" pod "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" (UID: "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:39.094870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.094831    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls" pod "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" (UID: "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:18:39.098971 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.098873    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s" pod "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" (UID: "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e"). InnerVolumeSpecName "kube-api-access-snz5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:39.099138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.099103    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs" pod "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" (UID: "b2bd1652-04f9-4b0b-8da8-4a9f537ef40e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:18:39.191101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.191044    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snz5s\" (UniqueName: \"kubernetes.io/projected/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-kube-api-access-snz5s\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:39.191101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.191082    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:39.191101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.191096    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:39.191101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.191106    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:39.470703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.470668    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:39.555402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.550542    5784 generic.go:356] "Generic (PLEG): container finished" podID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" containerID="6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c" exitCode=0
Nov 03 12:18:39.555402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.550812    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" event={"ID":"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e","Type":"ContainerDied","Data":"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"}
Nov 03 12:18:39.555402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.550904    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4" event={"ID":"b2bd1652-04f9-4b0b-8da8-4a9f537ef40e","Type":"ContainerDied","Data":"99452bad7863f3ced06514f41af16db707c984f832d950ba647cde0531fb26ca"}
Nov 03 12:18:39.555402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.550936    5784 scope.go:122] "RemoveContainer" containerID="6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"
Nov 03 12:18:39.555402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.551107    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"
Nov 03 12:18:39.647417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.638809    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:39.674596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.674433    5784 scope.go:122] "RemoveContainer" containerID="6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"
Nov 03 12:18:39.674865 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:39.674840    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c\": container with ID starting with 6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c not found: ID does not exist" containerID="6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"
Nov 03 12:18:39.674928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.674877    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c"} err="failed to get container status \"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c\": rpc error: code = NotFound desc = could not find container \"6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c\": container with ID starting with 6fb2ef760601689b5f75cdb2e343b3d1876fff436cffdac96954d2cd6293717c not found: ID does not exist"
Nov 03 12:18:39.772286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.772195    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:18:39.818303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:39.818267    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-931523-76b6f799b9-b5tt4"]
Nov 03 12:18:40.469596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:40.469506    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:41.032449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.032414    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:18:41.032932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.032481    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:18:41.091322 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.091230    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:41.466660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.466634    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:18:41.568178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568068    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568131    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568176    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568207    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568240    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568281    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568343    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568457    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.568663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.568526    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\") pod \"32ddae72-8c9f-46ea-9136-35b3f84289ba\" (UID: \"32ddae72-8c9f-46ea-9136-35b3f84289ba\") "
Nov 03 12:18:41.569154 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.569100    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.569328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.569303    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.569328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.569302    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:18:41.570218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.570182    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.570875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.570854    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.573764 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.573714    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:18:41.574846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.574793    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "kube-api-access-gtxfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:41.576703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.576665    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:18:41.577623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.577604    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy" pod "32ddae72-8c9f-46ea-9136-35b3f84289ba" (UID: "32ddae72-8c9f-46ea-9136-35b3f84289ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:18:41.594766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.591970    5784 generic.go:356] "Generic (PLEG): container finished" podID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerID="38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54" exitCode=0
Nov 03 12:18:41.594766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.592042    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" event={"ID":"32ddae72-8c9f-46ea-9136-35b3f84289ba","Type":"ContainerDied","Data":"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"}
Nov 03 12:18:41.594766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.592091    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n" event={"ID":"32ddae72-8c9f-46ea-9136-35b3f84289ba","Type":"ContainerDied","Data":"8f862446471a8cf105e0955bef41999d7220fd292ebf7ef515ce332c99f16317"}
Nov 03 12:18:41.594766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.592122    5784 scope.go:122] "RemoveContainer" containerID="38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"
Nov 03 12:18:41.594766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.592233    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"
Nov 03 12:18:41.616212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.616188    5784 scope.go:122] "RemoveContainer" containerID="38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"
Nov 03 12:18:41.616559 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:41.616531    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54\": container with ID starting with 38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54 not found: ID does not exist" containerID="38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"
Nov 03 12:18:41.616663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.616570    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54"} err="failed to get container status \"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54\": rpc error: code = NotFound desc = could not find container \"38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54\": container with ID starting with 38cf2689f18d324489d8d6f46003f1632e3a03025c58c79a37edaa098b204a54 not found: ID does not exist"
Nov 03 12:18:41.660411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.657216    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:41.669985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.669949    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670199    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670223    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670238    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670251    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxfv\" (UniqueName: \"kubernetes.io/projected/32ddae72-8c9f-46ea-9136-35b3f84289ba-kube-api-access-gtxfv\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670265    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670277    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32ddae72-8c9f-46ea-9136-35b3f84289ba-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670294    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32ddae72-8c9f-46ea-9136-35b3f84289ba-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.670415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.670309    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32ddae72-8c9f-46ea-9136-35b3f84289ba-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:18:41.685397 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.685351    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" path="/var/lib/kubelet/pods/b2bd1652-04f9-4b0b-8da8-4a9f537ef40e/volumes"
Nov 03 12:18:41.727148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.727114    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:18:41.731653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:41.731624    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-7fj6n"]
Nov 03 12:18:43.682513 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:43.682474    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" path="/var/lib/kubelet/pods/32ddae72-8c9f-46ea-9136-35b3f84289ba/volumes"
Nov 03 12:18:44.340831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:44.340796    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:44.981461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:44.981431    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:45.395087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:45.395052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:45.553219 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:45.553182    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:46.395648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:46.395618    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:46.559640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:46.559607    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:47.829578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:47.829545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:48.344046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:48.344007    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:49.515348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:49.515317    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:49.726272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:49.726232    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:49.927234 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:49.927201    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:50.079964 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:50.079920    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:50.257528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:50.254206    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:50.615249 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:50.615166    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:50.995496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:50.995460    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:52.626875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:52.626833    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:52.788554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:52.788524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:54.977919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:54.977888    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:55.241600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:55.241443    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:55.408288 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:55.408253    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:56.473945 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:18:56.473909    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:18:58.142293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:58.142261    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:58.303742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:58.303695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:18:59.992804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:18:59.992772    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:04.534617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:04.534568    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:06.430020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:06.429985    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:07.344767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:07.344728    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:07.642930 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:07.642900    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:07.803836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:07.803805    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:07.959504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:07.959368    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:08.139717 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:08.139585    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:09.472001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:09.471965    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:12.281589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.281558    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:19:12.282362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282338    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" containerName="maas-api"
Nov 03 12:19:12.282362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282360    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" containerName="maas-api"
Nov 03 12:19:12.282593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282459    5784 cpu_manager.go:414] "containerMap: removing container" podUID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" containerName="maas-api"
Nov 03 12:19:12.282593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282471    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" containerName="maas-api"
Nov 03 12:19:12.282593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282490    5784 cpu_manager.go:414] "containerMap: removing container" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy"
Nov 03 12:19:12.282593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282500    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy"
Nov 03 12:19:12.282805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282677    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bd1652-04f9-4b0b-8da8-4a9f537ef40e" containerName="maas-api"
Nov 03 12:19:12.282805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282703    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b5cdfc-2f58-44a8-a2c0-12e686fb563e" containerName="maas-api"
Nov 03 12:19:12.282805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.282712    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ddae72-8c9f-46ea-9136-35b3f84289ba" containerName="istio-proxy"
Nov 03 12:19:12.286589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.286567    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.307109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.307079    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:12.335771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.335744    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69fd8043-openshift-default-dockercfg-qfwfh\""
Nov 03 12:19:12.378006 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.377974    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378006 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378012    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfx5x\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378031    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378126    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378210    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378244    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378432    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378529    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.378610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.378583    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.402428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.402388    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:19:12.479712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479683    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.479879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479792    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.479879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479828    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfx5x\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.479879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479857    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479883    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479920    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.479966    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480095    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480189    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480260    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480492    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480569    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.480667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480601    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.481322 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.480955    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.482308 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.482287    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.482531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.482514    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.488251 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.488229    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:12.532759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.532690    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.532933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.532916    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfx5x\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:12.601586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:12.601550    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:13.024247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.024212    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:19:13.060429 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:19:13.060400    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404cf2d0_b1cb_423a_b79d_7b19819514b4.slice/crio-124f214bf5629470cc8f841fca29469d258717f0d5b9f18019e9022688e0cb07 WatchSource:0}: Error finding container 124f214bf5629470cc8f841fca29469d258717f0d5b9f18019e9022688e0cb07: Status 404 returned error can't find the container with id 124f214bf5629470cc8f841fca29469d258717f0d5b9f18019e9022688e0cb07
Nov 03 12:19:13.062838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.062802    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:13.062926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.062874    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:13.062926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.062903    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:13.949572 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.949536    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" event={"ID":"404cf2d0-b1cb-423a-b79d-7b19819514b4","Type":"ContainerStarted","Data":"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"}
Nov 03 12:19:13.949982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:13.949598    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" event={"ID":"404cf2d0-b1cb-423a-b79d-7b19819514b4","Type":"ContainerStarted","Data":"124f214bf5629470cc8f841fca29469d258717f0d5b9f18019e9022688e0cb07"}
Nov 03 12:19:14.106550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:14.106450    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podStartSLOduration=2.106429411 podStartE2EDuration="2.106429411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:14.103824694 +0000 UTC m=+1660.546027000" watchObservedRunningTime="2026-11-03 12:19:14.106429411 +0000 UTC m=+1660.548631717"
Nov 03 12:19:14.368019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:14.367939    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:14.602166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:14.602128    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:14.603805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:14.603779    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:14.603932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:14.603831    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:15.602008 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:15.601975    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:15.602451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:15.602030    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:16.602162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:16.602126    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:16.602573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:16.602172    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:17.046481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:17.043949    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:17.202749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:17.201512    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:17.604053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:17.603994    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:17.604763 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:17.604692    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:18.340313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:18.340259    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:18.602734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:18.602644    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:18.602734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:18.602707    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:19.331642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:19.331575    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:19.602088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:19.602009    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 12:19:19.602088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:19.602066    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 12:19:19.726185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:19.726139    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:20.607552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:20.607518    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:20.607951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:20.607928    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:20.608790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:20.608760    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:19:24.327780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:24.327713    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:25.453196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:25.453162    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:25.606196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:25.606162    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:25.759591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:25.759504    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:26.508107 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:19:26.508066    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:19:27.239368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.239328    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:19:27.243829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.243807    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285698 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285650    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285718    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285773    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285811    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285846    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.285901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285872    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.286181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.285966    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ndr\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.286181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.286015    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.286181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.286074    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.344890 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.344851    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:19:27.387068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387030    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387093    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387132    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387177    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387230    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387269    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387299    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387584    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387434    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387601    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387736    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ndr\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387829    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.387956 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.387917    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.388289 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.388270    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.389711 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.389691    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.389912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.389890    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.439825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.439791    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.439976 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.439927    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ndr\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.559436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.559333    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:27.992100 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.992069    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:19:27.992852 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:19:27.992827    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228facfc_1620_410f_b6f3_51aed832c6ab.slice/crio-29ac9998a209ea06db32fe99175593570e82cbc2a517badcdded69a5abe8af47 WatchSource:0}: Error finding container 29ac9998a209ea06db32fe99175593570e82cbc2a517badcdded69a5abe8af47: Status 404 returned error can't find the container with id 29ac9998a209ea06db32fe99175593570e82cbc2a517badcdded69a5abe8af47
Nov 03 12:19:27.995329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.995292    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:27.995440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.995358    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:27.995440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:27.995424    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:28.062655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:28.062629    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" event={"ID":"228facfc-1620-410f-b6f3-51aed832c6ab","Type":"ContainerStarted","Data":"29ac9998a209ea06db32fe99175593570e82cbc2a517badcdded69a5abe8af47"}
Nov 03 12:19:28.174431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:28.174366    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podStartSLOduration=1.174350547 podStartE2EDuration="1.174350547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:28.169009837 +0000 UTC m=+1674.611212161" watchObservedRunningTime="2026-11-03 12:19:28.174350547 +0000 UTC m=+1674.616552851"
Nov 03 12:19:28.560335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:28.560244    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:29.072609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:29.072566    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" event={"ID":"228facfc-1620-410f-b6f3-51aed832c6ab","Type":"ContainerStarted","Data":"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"}
Nov 03 12:19:29.171656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:29.171622    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:29.561591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:29.561556    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:29.562013 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:29.561621    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:30.560690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:30.560655    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:30.560886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:30.560711    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:31.446329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:31.446297    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:31.560183 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:31.560143    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:31.560355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:31.560199    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:32.560134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:32.560096    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:32.560601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:32.560163    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:33.560615 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:33.560577    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:33.561043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:33.560646    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:34.166680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:34.166644    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:34.560444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:34.560365    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 12:19:34.560604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:34.560485    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 12:19:35.565364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:35.565314    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:35.565807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:35.565629    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:35.566428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:35.566411    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:19:38.770288 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.770256    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:19:38.775740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.775721    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.827336 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.827301    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:19:38.866091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.866055    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-646ch\" (UniqueName: \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.866277 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.866148    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.866345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.866278    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.866633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.866609    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.868982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.868955    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-deb666-serving-cert\""
Nov 03 12:19:38.968333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.968293    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.968913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.968442    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.968913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.968479    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-646ch\" (UniqueName: \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.968913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.968502    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.968913 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:19:38.968519    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-serving-cert: secret "maas-api-e2e-ipp-a-deb666-serving-cert" not found
Nov 03 12:19:38.968913 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:19:38.968597    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls podName:5ab47a9e-ce12-4d68-9c30-ebec8846a641 nodeName:}" failed. No retries permitted until 2026-11-03 12:19:39.468575281 +0000 UTC m=+1685.910777578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls") pod "maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" (UID: "5ab47a9e-ce12-4d68-9c30-ebec8846a641") : secret "maas-api-e2e-ipp-a-deb666-serving-cert" not found
Nov 03 12:19:38.969411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.969351    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.976256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.976230    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:38.981770 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:38.981734    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:39.021847 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.021777    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-646ch\" (UniqueName: \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:39.134192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.134154    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:39.294760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.294678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:39.489115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.489068    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:39.492031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.491998    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") pod \"maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:39.690664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:39.690638    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:40.086420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:40.082890    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:19:40.086420 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:19:40.083060    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab47a9e_ce12_4d68_9c30_ebec8846a641.slice/crio-2256be0f97ba57142d4d0794b51b440400973a8d30c0a02f9868756e5343ffda WatchSource:0}: Error finding container 2256be0f97ba57142d4d0794b51b440400973a8d30c0a02f9868756e5343ffda: Status 404 returned error can't find the container with id 2256be0f97ba57142d4d0794b51b440400973a8d30c0a02f9868756e5343ffda
Nov 03 12:19:40.086420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:40.085077    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:19:40.237441 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:40.237408    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" event={"ID":"5ab47a9e-ce12-4d68-9c30-ebec8846a641","Type":"ContainerStarted","Data":"2256be0f97ba57142d4d0794b51b440400973a8d30c0a02f9868756e5343ffda"}
Nov 03 12:19:40.897278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:40.897239    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:41.247307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:41.247216    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" event={"ID":"5ab47a9e-ce12-4d68-9c30-ebec8846a641","Type":"ContainerStarted","Data":"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"}
Nov 03 12:19:41.247734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:41.247323    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:41.355843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:41.355792    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" podStartSLOduration=2.543386926 podStartE2EDuration="3.355778919s" totalImagesPullingTime="812.391993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:41.352898137 +0000 UTC m=+1687.795100454" watchObservedRunningTime="2026-11-03 12:19:41.355778919 +0000 UTC m=+1687.797981224"
Nov 03 12:19:42.230170 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.230130    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:19:42.235424 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.235395    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.334417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.334362    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:19:42.335242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335208    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335291    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335325    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335449    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpgn\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335589    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335685    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335719    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335803    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.336071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.335856    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437414    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437470    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437548    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437590    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437617    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437673    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437704    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437781    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpgn\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.437981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.437901    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.438108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.438082    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.438176 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.438095    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.438276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.438256    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.438414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.438358    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.438466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.438429    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.439791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.439770    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.440074 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.440058    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.490309 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.490237    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.490309 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.490286    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpgn\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\") pod \"e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") " pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.549782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.549753    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:42.977421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.977368    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:19:42.977742 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:19:42.977712    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd81f525b_a10f_4025_80d2_56959593edd1.slice/crio-38003939bc85941ab8e42d9cd81816f2ee19fe08151a31f1697202cde53c36e4 WatchSource:0}: Error finding container 38003939bc85941ab8e42d9cd81816f2ee19fe08151a31f1697202cde53c36e4: Status 404 returned error can't find the container with id 38003939bc85941ab8e42d9cd81816f2ee19fe08151a31f1697202cde53c36e4
Nov 03 12:19:42.980458 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.980418    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:42.980566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.980504    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:42.980566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:42.980553    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:19:43.262733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:43.262650    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" event={"ID":"d81f525b-a10f-4025-80d2-56959593edd1","Type":"ContainerStarted","Data":"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"}
Nov 03 12:19:43.262733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:43.262699    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" event={"ID":"d81f525b-a10f-4025-80d2-56959593edd1","Type":"ContainerStarted","Data":"38003939bc85941ab8e42d9cd81816f2ee19fe08151a31f1697202cde53c36e4"}
Nov 03 12:19:43.371615 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:43.371562    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podStartSLOduration=1.371546255 podStartE2EDuration="1.371546255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:19:43.37029456 +0000 UTC m=+1689.812496893" watchObservedRunningTime="2026-11-03 12:19:43.371546255 +0000 UTC m=+1689.813748559"
Nov 03 12:19:43.552648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:43.552543    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:43.635404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:43.631092    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:44.552038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:44.551999    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 12:19:44.552683 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:44.552646    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 12:19:45.550923 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:45.550879    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 12:19:45.551080 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:45.550942    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 12:19:45.941313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:45.941270    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:46.551738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:46.551706    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 12:19:46.551935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:46.551774    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 12:19:47.260138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:47.260109    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:19:47.550718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:47.550629    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 12:19:47.550718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:47.550680    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 12:19:48.062036 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:48.062001    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:48.223429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:48.219762    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:48.376477 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:48.376445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:48.550201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:48.550153    5784 patch_prober.go:29] interesting pod/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 12:19:48.550393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:48.550217    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 12:19:49.581899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:49.581854    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:49.582602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:49.582575    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:49.586501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:49.586477    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:19:49.822507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:49.822420    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:52.008273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.008240    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:19:52.008673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.008464    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerName="maas-api" containerID="cri-o://9a2047209f3e0dc35764839399b7b33b44e23bd073b6b7a4ac214f0f88581cc4" gracePeriod=30
Nov 03 12:19:52.147574 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.147496    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:52.179760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.179545    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.91:8443/health\": dial tcp 10.134.0.91:8443: connect: connection refused"
Nov 03 12:19:52.354505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.354468    5784 generic.go:356] "Generic (PLEG): container finished" podID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerID="9a2047209f3e0dc35764839399b7b33b44e23bd073b6b7a4ac214f0f88581cc4" exitCode=0
Nov 03 12:19:52.354665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.354644    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" event={"ID":"debbe158-c45b-4d7f-a6d7-00d029742d85","Type":"ContainerDied","Data":"9a2047209f3e0dc35764839399b7b33b44e23bd073b6b7a4ac214f0f88581cc4"}
Nov 03 12:19:52.414582 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.414546    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:19:52.499069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.499034    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") pod \"debbe158-c45b-4d7f-a6d7-00d029742d85\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") "
Nov 03 12:19:52.499829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.499807    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\" (UniqueName: \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\") pod \"debbe158-c45b-4d7f-a6d7-00d029742d85\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") "
Nov 03 12:19:52.500515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.500494    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\") pod \"debbe158-c45b-4d7f-a6d7-00d029742d85\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") "
Nov 03 12:19:52.501579 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.501463    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca" pod "debbe158-c45b-4d7f-a6d7-00d029742d85" (UID: "debbe158-c45b-4d7f-a6d7-00d029742d85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:19:52.501856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.501837    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\") pod \"debbe158-c45b-4d7f-a6d7-00d029742d85\" (UID: \"debbe158-c45b-4d7f-a6d7-00d029742d85\") "
Nov 03 12:19:52.503246 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.503224    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/debbe158-c45b-4d7f-a6d7-00d029742d85-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:52.503515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.503300    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2" pod "debbe158-c45b-4d7f-a6d7-00d029742d85" (UID: "debbe158-c45b-4d7f-a6d7-00d029742d85"). InnerVolumeSpecName "kube-api-access-f59d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:19:52.506767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.506733    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs" pod "debbe158-c45b-4d7f-a6d7-00d029742d85" (UID: "debbe158-c45b-4d7f-a6d7-00d029742d85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:19:52.507877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.507853    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls" pod "debbe158-c45b-4d7f-a6d7-00d029742d85" (UID: "debbe158-c45b-4d7f-a6d7-00d029742d85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:19:52.605487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.605373    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:52.605487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.605433    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/debbe158-c45b-4d7f-a6d7-00d029742d85-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:52.605487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:52.605447    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59d2\" (UniqueName: \"kubernetes.io/projected/debbe158-c45b-4d7f-a6d7-00d029742d85-kube-api-access-f59d2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:19:53.364212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.364168    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55" event={"ID":"debbe158-c45b-4d7f-a6d7-00d029742d85","Type":"ContainerDied","Data":"347fc61ff91a2577cc755b766c9f9c9540d9d4659fecf7ca67526528eeeb31ec"}
Nov 03 12:19:53.364212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.364192    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"
Nov 03 12:19:53.364753 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.364243    5784 scope.go:122] "RemoveContainer" containerID="9a2047209f3e0dc35764839399b7b33b44e23bd073b6b7a4ac214f0f88581cc4"
Nov 03 12:19:53.517057 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.517013    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:19:53.562635 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.562604    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f6c034-f6bd8454c-vqd55"]
Nov 03 12:19:53.684486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.684453    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" path="/var/lib/kubelet/pods/debbe158-c45b-4d7f-a6d7-00d029742d85/volumes"
Nov 03 12:19:53.913679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:53.913644    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:54.088890 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:54.088804    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:54.257539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:54.257508    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:54.405041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:54.405008    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:55.515580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:55.515477    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:55.997826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:55.997788    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:56.474778 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:19:56.474738    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:19:56.766397 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:56.766308    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:57.605716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:57.605685    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:19:58.112276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:19:58.112241    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:00.597212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:00.597177    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:02.219079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:02.219043    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:02.374654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:02.374613    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:02.602080 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:02.601995    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:04.564611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:04.564575    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:20:04.565122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:04.564843    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" podUID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" containerName="maas-api" containerID="cri-o://597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93" gracePeriod=30
Nov 03 12:20:04.687496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:04.687463    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:05.095649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.095617    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:20:05.253835 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.253796    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\" (UniqueName: \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\") pod \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") "
Nov 03 12:20:05.254147 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.254123    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\") pod \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") "
Nov 03 12:20:05.254508 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.254476    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca" pod "23b90192-c3e5-4453-b5a9-81b1d9467e5a" (UID: "23b90192-c3e5-4453-b5a9-81b1d9467e5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:20:05.254643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.254510    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") pod \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") "
Nov 03 12:20:05.254802 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.254784    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\") pod \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\" (UID: \"23b90192-c3e5-4453-b5a9-81b1d9467e5a\") "
Nov 03 12:20:05.255521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.255463    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23b90192-c3e5-4453-b5a9-81b1d9467e5a-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:05.256640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.256610    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn" pod "23b90192-c3e5-4453-b5a9-81b1d9467e5a" (UID: "23b90192-c3e5-4453-b5a9-81b1d9467e5a"). InnerVolumeSpecName "kube-api-access-jshsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:20:05.258314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.258283    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs" pod "23b90192-c3e5-4453-b5a9-81b1d9467e5a" (UID: "23b90192-c3e5-4453-b5a9-81b1d9467e5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:20:05.258531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.258511    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls" pod "23b90192-c3e5-4453-b5a9-81b1d9467e5a" (UID: "23b90192-c3e5-4453-b5a9-81b1d9467e5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:20:05.356929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.356875    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:05.356929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.356914    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b90192-c3e5-4453-b5a9-81b1d9467e5a-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:05.356929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.356924    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshsn\" (UniqueName: \"kubernetes.io/projected/23b90192-c3e5-4453-b5a9-81b1d9467e5a-kube-api-access-jshsn\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:20:05.469710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.469673    5784 generic.go:356] "Generic (PLEG): container finished" podID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" containerID="597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93" exitCode=0
Nov 03 12:20:05.469900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.469714    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" event={"ID":"23b90192-c3e5-4453-b5a9-81b1d9467e5a","Type":"ContainerDied","Data":"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"}
Nov 03 12:20:05.469900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.469731    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"
Nov 03 12:20:05.469900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.469779    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm" event={"ID":"23b90192-c3e5-4453-b5a9-81b1d9467e5a","Type":"ContainerDied","Data":"47675088e8d82e22922be97f609f0b0eedc238f66ab61197268934205315ea7d"}
Nov 03 12:20:05.469900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.469812    5784 scope.go:122] "RemoveContainer" containerID="597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"
Nov 03 12:20:05.480286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.480267    5784 scope.go:122] "RemoveContainer" containerID="597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"
Nov 03 12:20:05.480548 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:20:05.480527    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93\": container with ID starting with 597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93 not found: ID does not exist" containerID="597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"
Nov 03 12:20:05.480622 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.480561    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93"} err="failed to get container status \"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93\": rpc error: code = NotFound desc = could not find container \"597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93\": container with ID starting with 597d5322bf245966eedb99c07baaa994131c8f0dac5ca0356cf18a344e0a8c93 not found: ID does not exist"
Nov 03 12:20:05.625427 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.625391    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:20:05.671479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.671447    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-89a88d-56467c7f6b-4czrm"]
Nov 03 12:20:05.685329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:05.685293    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" path="/var/lib/kubelet/pods/23b90192-c3e5-4453-b5a9-81b1d9467e5a/volumes"
Nov 03 12:20:06.069321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.069268    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:20:06.070066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070044    5784 cpu_manager.go:414] "containerMap: removing container" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerName="maas-api"
Nov 03 12:20:06.070066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070065    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerName="maas-api"
Nov 03 12:20:06.070210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070103    5784 cpu_manager.go:414] "containerMap: removing container" podUID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" containerName="maas-api"
Nov 03 12:20:06.070210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070108    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" containerName="maas-api"
Nov 03 12:20:06.070210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070200    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b90192-c3e5-4453-b5a9-81b1d9467e5a" containerName="maas-api"
Nov 03 12:20:06.070300 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.070213    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="debbe158-c45b-4d7f-a6d7-00d029742d85" containerName="maas-api"
Nov 03 12:20:06.074758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.074739    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.122679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.122646    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-45b1b7-serving-cert\""
Nov 03 12:20:06.124990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.124959    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:20:06.172725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.172691    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.172915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.172753    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.172966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.172941    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qpn\" (UniqueName: \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.173068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.173051    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.271034 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.270999    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:06.274447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.274417    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qpn\" (UniqueName: \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.274576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.274521    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.274620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.274601    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.274688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.274674    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.274747 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:20:06.274728    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-serving-cert: secret "maas-api-e2e-ipp-b-45b1b7-serving-cert" not found
Nov 03 12:20:06.274816 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:20:06.274801    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls podName:c1ba45b6-b88f-4dfb-80b5-30baef2925ae nodeName:}" failed. No retries permitted until 2026-11-03 12:20:06.774780289 +0000 UTC m=+1713.216982582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls") pod "maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" (UID: "c1ba45b6-b88f-4dfb-80b5-30baef2925ae") : secret "maas-api-e2e-ipp-b-45b1b7-serving-cert" not found
Nov 03 12:20:06.275323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.275304    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.277066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.277046    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.330971 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.330886    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qpn\" (UniqueName: \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.425756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.425726    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:06.578826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.578792    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:06.782736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.782693    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.785354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.785330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") pod \"maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:06.988587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:06.988548    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:07.364082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:07.364049    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:20:07.365838 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:20:07.365811    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ba45b6_b88f_4dfb_80b5_30baef2925ae.slice/crio-9fc355fb5e883f507cf9b5b7dd4414ca187ddadb7ed645bc287a2844d4ba8261 WatchSource:0}: Error finding container 9fc355fb5e883f507cf9b5b7dd4414ca187ddadb7ed645bc287a2844d4ba8261: Status 404 returned error can't find the container with id 9fc355fb5e883f507cf9b5b7dd4414ca187ddadb7ed645bc287a2844d4ba8261
Nov 03 12:20:07.498341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:07.498302    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" event={"ID":"c1ba45b6-b88f-4dfb-80b5-30baef2925ae","Type":"ContainerStarted","Data":"9fc355fb5e883f507cf9b5b7dd4414ca187ddadb7ed645bc287a2844d4ba8261"}
Nov 03 12:20:08.026544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:08.026508    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:08.194153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:08.194119    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:08.507072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:08.507033    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" event={"ID":"c1ba45b6-b88f-4dfb-80b5-30baef2925ae","Type":"ContainerStarted","Data":"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"}
Nov 03 12:20:08.507255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:08.507101    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:08.615078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:08.614977    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" podStartSLOduration=1.7326360140000001 podStartE2EDuration="2.61496281s" totalImagesPullingTime="882.326796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:20:08.612857883 +0000 UTC m=+1715.055060207" watchObservedRunningTime="2026-11-03 12:20:08.61496281 +0000 UTC m=+1715.057165115"
Nov 03 12:20:09.494848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:09.494819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:09.967616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:09.967577    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:10.904383 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:10.904337    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:11.233990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:11.233873    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:11.961554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:11.961511    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:12.186807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:12.186771    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:14.523705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:14.523675    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:20:15.856299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:15.856260    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:16.003351 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:16.003320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:16.162875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:16.162831    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:16.323677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:16.323645    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:19.056439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:19.056370    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:19.215753 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:19.215722    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:20.732551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:20.732515    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:22.820393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:22.820350    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:23.448839 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:23.448801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:24.378110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:24.378071    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:24.528005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:24.527972    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:25.612520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:25.612481    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:26.482746 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:20:26.482697    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:20:28.832603 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:28.832538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:30.349541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:30.349496    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:35.256024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:35.255972    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:40.161323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:40.161287    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:41.732960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:41.732921    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:41.961895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:41.961857    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:42.135856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:42.135822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:45.104251 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:45.104215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:50.291364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:50.291333    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:50.444216 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:50.444185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:52.788723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:52.788686    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:53.683666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:53.683629    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:53.933628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:53.933592    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:54.090832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:54.090756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:54.255578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:54.255537    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:54.471710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:54.471677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:56.476789 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:20:56.476746    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:20:56.708347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:56.708318    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:59.013680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:59.013650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:59.186575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:59.186539    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:20:59.355259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:20:59.355172    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:00.245203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:00.245168    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:01.414917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:01.414882    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:02.062880 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:02.062845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:02.246440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:02.240409    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:06.348157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:06.348117    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:07.108594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:07.108558    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:07.266706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:07.266671    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:09.882104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:09.882071    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:10.473943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:10.473911    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:10.632475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:10.632445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:10.983165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:10.983127    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:11.128522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:11.128485    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:11.282901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:11.282812    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:15.917842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:15.917806    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:20.469715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:20.469680    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:24.897486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:24.897447    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:26.473915 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:21:26.473878    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:21:29.522327 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:29.522283    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:32.825327 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:32.825262    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:33.268781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:33.268743    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:34.007474 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:34.007445    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:21:34.015427 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:34.015360    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:21:35.765161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:35.764296    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:40.369531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:40.369492    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:43.694321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:43.694020    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:44.094930 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:44.094852    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:46.714264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:46.714224    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:48.377743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:48.377709    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:49.596172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:49.596137    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:50.181822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:50.181786    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:50.614205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:50.614039    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:50.765664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:50.765603    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:52.589222 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:52.589176    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:52.743986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:52.743951    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:53.021261 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:53.021224    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:54.245757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:54.245726    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:54.423697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:54.423646    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:54.610975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:54.610874    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:56.473131 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:21:56.473097    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:21:57.333873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:57.333822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:57.496101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:57.496065    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:21:57.681833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:21:57.680862    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:02.387036 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:02.387003    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:05.665341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:05.665298    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:06.125119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:06.125026    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:08.553937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:08.553895    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:13.024641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:13.024602    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:17.512220 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:17.512171    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:18.892677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:18.892640    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:19.334881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:19.334782    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:19.483818 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:19.483756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:21.641998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:21.641958    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:23.265645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.265604    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:23.424758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.424727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:23.727896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.727856    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:22:23.728120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.728093    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy" containerID="cri-o://da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791" gracePeriod=30
Nov 03 12:22:23.745941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.745901    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:22:23.746199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.746152    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy" containerID="cri-o://bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb" gracePeriod=30
Nov 03 12:22:23.762266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.762217    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:22:23.763290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.762508    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy" containerID="cri-o://4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7" gracePeriod=30
Nov 03 12:22:23.765617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:23.765562    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:24.271370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:24.271338    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:24.442874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:24.442845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:25.938404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:25.938360    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:26.203732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:26.203652    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:26.351779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:26.351739    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:26.472354 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:22:26.472269    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:22:28.030665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:28.030632    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:29.054279 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.054247    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:22:29.081749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.081727    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:22:29.122443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.122417    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:22:29.149561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149533    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.149727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149585    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149615    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149651    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.149727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149676    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.149727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149706    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149752    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149793    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149826    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149873    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149899    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149932    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149957    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.149991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.149985    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150012    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150045    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150081    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150121    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150153    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150225    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150212    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150263    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150331    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150410    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150488    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150548    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150578    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\") pod \"228facfc-1620-410f-b6f3-51aed832c6ab\" (UID: \"228facfc-1620-410f-b6f3-51aed832c6ab\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150608    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150644    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150678    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150688    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\") pod \"d81f525b-a10f-4025-80d2-56959593edd1\" (UID: \"d81f525b-a10f-4025-80d2-56959593edd1\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150745    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\") pod \"404cf2d0-b1cb-423a-b79d-7b19819514b4\" (UID: \"404cf2d0-b1cb-423a-b79d-7b19819514b4\") "
Nov 03 12:22:29.150917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.150895    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.151412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151271    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.151611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151589    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d81f525b-a10f-4025-80d2-56959593edd1-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151616    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151633    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151649    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151663    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151678    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/228facfc-1620-410f-b6f3-51aed832c6ab-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151691    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.151945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151813    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:22:29.152328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151892    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.152328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.151936    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.152871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.152570    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.152871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.152585    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.153975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.153401    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.153975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.153568    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.153975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.153921    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.154155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.154065    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.154530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.154471    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.154613 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.154576    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "kube-api-access-r8ndr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.156186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.156161    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.164804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164779    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.164897 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164783    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.164897 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164806    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "kube-api-access-kdpgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.164897 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164885    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.165015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164900    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.165015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164924    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy" pod "228facfc-1620-410f-b6f3-51aed832c6ab" (UID: "228facfc-1620-410f-b6f3-51aed832c6ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:22:29.165015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.164941    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo" pod "d81f525b-a10f-4025-80d2-56959593edd1" (UID: "d81f525b-a10f-4025-80d2-56959593edd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:22:29.166093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.166070    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x" pod "404cf2d0-b1cb-423a-b79d-7b19819514b4" (UID: "404cf2d0-b1cb-423a-b79d-7b19819514b4"). InnerVolumeSpecName "kube-api-access-jfx5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255770    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255807    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/228facfc-1620-410f-b6f3-51aed832c6ab-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255821    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255836    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255850    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.255921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255864    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ndr\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-kube-api-access-r8ndr\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.255880    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256434    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256451    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404cf2d0-b1cb-423a-b79d-7b19819514b4-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256465    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256498    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256516    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d81f525b-a10f-4025-80d2-56959593edd1-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256532    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpgn\" (UniqueName: \"kubernetes.io/projected/d81f525b-a10f-4025-80d2-56959593edd1-kube-api-access-kdpgn\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256549    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256563    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d81f525b-a10f-4025-80d2-56959593edd1-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256577    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/228facfc-1620-410f-b6f3-51aed832c6ab-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256591    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfx5x\" (UniqueName: \"kubernetes.io/projected/404cf2d0-b1cb-423a-b79d-7b19819514b4-kube-api-access-jfx5x\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256606    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404cf2d0-b1cb-423a-b79d-7b19819514b4-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256621    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/228facfc-1620-410f-b6f3-51aed832c6ab-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.256660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.256635    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404cf2d0-b1cb-423a-b79d-7b19819514b4-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:22:29.795524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.795489    5784 generic.go:356] "Generic (PLEG): container finished" podID="d81f525b-a10f-4025-80d2-56959593edd1" containerID="bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb" exitCode=0
Nov 03 12:22:29.795716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.795646    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" event={"ID":"d81f525b-a10f-4025-80d2-56959593edd1","Type":"ContainerDied","Data":"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"}
Nov 03 12:22:29.795791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.795737    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms" event={"ID":"d81f525b-a10f-4025-80d2-56959593edd1","Type":"ContainerDied","Data":"38003939bc85941ab8e42d9cd81816f2ee19fe08151a31f1697202cde53c36e4"}
Nov 03 12:22:29.795847 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.795794    5784 scope.go:122] "RemoveContainer" containerID="bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"
Nov 03 12:22:29.796026 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.796006    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"
Nov 03 12:22:29.800642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.800616    5784 generic.go:356] "Generic (PLEG): container finished" podID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerID="4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7" exitCode=0
Nov 03 12:22:29.800766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.800710    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"
Nov 03 12:22:29.800766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.800728    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" event={"ID":"404cf2d0-b1cb-423a-b79d-7b19819514b4","Type":"ContainerDied","Data":"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"}
Nov 03 12:22:29.800853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.800767    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm" event={"ID":"404cf2d0-b1cb-423a-b79d-7b19819514b4","Type":"ContainerDied","Data":"124f214bf5629470cc8f841fca29469d258717f0d5b9f18019e9022688e0cb07"}
Nov 03 12:22:29.804007 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.803978    5784 generic.go:356] "Generic (PLEG): container finished" podID="228facfc-1620-410f-b6f3-51aed832c6ab" containerID="da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791" exitCode=0
Nov 03 12:22:29.804157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.804044    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" event={"ID":"228facfc-1620-410f-b6f3-51aed832c6ab","Type":"ContainerDied","Data":"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"}
Nov 03 12:22:29.804157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.804050    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"
Nov 03 12:22:29.804157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.804084    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s" event={"ID":"228facfc-1620-410f-b6f3-51aed832c6ab","Type":"ContainerDied","Data":"29ac9998a209ea06db32fe99175593570e82cbc2a517badcdded69a5abe8af47"}
Nov 03 12:22:29.811245 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.811228    5784 scope.go:122] "RemoveContainer" containerID="bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"
Nov 03 12:22:29.811551 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:22:29.811531    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb\": container with ID starting with bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb not found: ID does not exist" containerID="bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"
Nov 03 12:22:29.811618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.811558    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb"} err="failed to get container status \"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb\": rpc error: code = NotFound desc = could not find container \"bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb\": container with ID starting with bf704bb31e1be1c9e9a4153a7d3fbd632bd29144837132c216a715464bcc49eb not found: ID does not exist"
Nov 03 12:22:29.811618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.811574    5784 scope.go:122] "RemoveContainer" containerID="4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"
Nov 03 12:22:29.820846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.820819    5784 scope.go:122] "RemoveContainer" containerID="4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"
Nov 03 12:22:29.821196 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:22:29.821160    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7\": container with ID starting with 4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7 not found: ID does not exist" containerID="4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"
Nov 03 12:22:29.821250 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.821218    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7"} err="failed to get container status \"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7\": rpc error: code = NotFound desc = could not find container \"4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7\": container with ID starting with 4986a65ccf7d4aca62be43484523352711bdae15fd7cae88c7f1d9c918d5dea7 not found: ID does not exist"
Nov 03 12:22:29.821295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.821259    5784 scope.go:122] "RemoveContainer" containerID="da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"
Nov 03 12:22:29.832106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.832088    5784 scope.go:122] "RemoveContainer" containerID="da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"
Nov 03 12:22:29.832478 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:22:29.832449    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791\": container with ID starting with da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791 not found: ID does not exist" containerID="da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"
Nov 03 12:22:29.832572 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.832477    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791"} err="failed to get container status \"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791\": rpc error: code = NotFound desc = could not find container \"da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791\": container with ID starting with da9a0df30e77a1f867e603b25af40d337aa6435af041818ee10bd38bcbbb2791 not found: ID does not exist"
Nov 03 12:22:29.949449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.949338    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:22:29.955059 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:29.955031    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-gjpms"]
Nov 03 12:22:30.056173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:30.056132    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:22:30.061999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:30.061974    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-8vhqm"]
Nov 03 12:22:30.163731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:30.163683    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:22:30.166949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:30.166923    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fd8043-openshift-default-67cb4c5745-7fl4s"]
Nov 03 12:22:30.953988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:30.953946    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:31.124306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:31.124251    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:31.683409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:31.683355    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" path="/var/lib/kubelet/pods/228facfc-1620-410f-b6f3-51aed832c6ab/volumes"
Nov 03 12:22:31.683882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:31.683867    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" path="/var/lib/kubelet/pods/404cf2d0-b1cb-423a-b79d-7b19819514b4/volumes"
Nov 03 12:22:31.684284 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:31.684272    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d81f525b-a10f-4025-80d2-56959593edd1" path="/var/lib/kubelet/pods/d81f525b-a10f-4025-80d2-56959593edd1/volumes"
Nov 03 12:22:32.178870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:32.178840    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:32.779315 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:32.779281    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:36.228828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:36.228775    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:39.078459 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:39.077995    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:43.818525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:43.818491    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:48.602882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:48.602842    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:49.175514 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:49.175483    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:49.654469 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:49.654430    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:49.803016 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:49.802968    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:50.775796 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:50.775763    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:50.925020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:50.924984    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:51.071686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:51.071594    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:54.287527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:54.287489    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:54.986913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:54.986879    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:55.146652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:55.146611    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:55.463934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:55.463902    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:56.472014 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:22:56.471979    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:22:56.607734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:56.607696    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:57.180521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:57.180487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:22:57.978129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:22:57.978093    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:00.216595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:00.216566    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:01.271666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:01.271629    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:02.031279 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:02.031238    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:02.179130 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:02.179091    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:02.855651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:02.855608    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:03.729388 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:03.729334    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:04.336018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.335979    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:23:04.336871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.336830    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerName="maas-api" containerID="cri-o://c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806" gracePeriod=30
Nov 03 12:23:04.462121 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.462085    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:04.696758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.696451    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:23:04.715560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.715507    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\" (UniqueName: \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\") pod \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") "
Nov 03 12:23:04.715560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.715560    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\") pod \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") "
Nov 03 12:23:04.715805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.715778    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\") pod \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") "
Nov 03 12:23:04.715859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.715818    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") pod \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\" (UID: \"c1ba45b6-b88f-4dfb-80b5-30baef2925ae\") "
Nov 03 12:23:04.716292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.716262    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca" pod "c1ba45b6-b88f-4dfb-80b5-30baef2925ae" (UID: "c1ba45b6-b88f-4dfb-80b5-30baef2925ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:04.718898 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.718849    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs" pod "c1ba45b6-b88f-4dfb-80b5-30baef2925ae" (UID: "c1ba45b6-b88f-4dfb-80b5-30baef2925ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:23:04.719019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.718962    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn" pod "c1ba45b6-b88f-4dfb-80b5-30baef2925ae" (UID: "c1ba45b6-b88f-4dfb-80b5-30baef2925ae"). InnerVolumeSpecName "kube-api-access-f4qpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:04.719320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.719298    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls" pod "c1ba45b6-b88f-4dfb-80b5-30baef2925ae" (UID: "c1ba45b6-b88f-4dfb-80b5-30baef2925ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:23:04.824457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.818583    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:04.824457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.818625    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4qpn\" (UniqueName: \"kubernetes.io/projected/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-kube-api-access-f4qpn\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:04.824457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.818641    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:04.824457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:04.818655    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1ba45b6-b88f-4dfb-80b5-30baef2925ae-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:05.071360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.071275    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:05.182752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.182717    5784 generic.go:356] "Generic (PLEG): container finished" podID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerID="c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806" exitCode=0
Nov 03 12:23:05.182935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.182775    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"
Nov 03 12:23:05.182935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.182811    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" event={"ID":"c1ba45b6-b88f-4dfb-80b5-30baef2925ae","Type":"ContainerDied","Data":"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"}
Nov 03 12:23:05.182935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.182868    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" event={"ID":"c1ba45b6-b88f-4dfb-80b5-30baef2925ae","Type":"ContainerDied","Data":"9fc355fb5e883f507cf9b5b7dd4414ca187ddadb7ed645bc287a2844d4ba8261"}
Nov 03 12:23:05.182935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.182899    5784 scope.go:122] "RemoveContainer" containerID="c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"
Nov 03 12:23:05.194537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.194518    5784 scope.go:122] "RemoveContainer" containerID="c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"
Nov 03 12:23:05.194829 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:23:05.194812    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806\": container with ID starting with c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806 not found: ID does not exist" containerID="c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"
Nov 03 12:23:05.194872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.194839    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806"} err="failed to get container status \"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806\": rpc error: code = NotFound desc = could not find container \"c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806\": container with ID starting with c160b3c52992849b5c72c66ca64d7a8ed8b747a197edbbd09f2f3cdc7de4e806 not found: ID does not exist"
Nov 03 12:23:05.220348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.220316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:05.337691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.337600    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:23:05.341662 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.341634    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf"]
Nov 03 12:23:05.682455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:05.682422    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" path="/var/lib/kubelet/pods/c1ba45b6-b88f-4dfb-80b5-30baef2925ae/volumes"
Nov 03 12:23:06.409551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:06.409519    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:07.521952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:07.521903    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-45b1b7-c6b76d8d-mqdwf" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.98:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:08.162233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:08.162187    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:23:08.162506 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:08.162476    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" containerID="cri-o://56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799" gracePeriod=30
Nov 03 12:23:08.924934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:08.924897    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:09.077229 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:09.077199    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:09.460909 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:09.459356    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:09.627848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:09.627812    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:10.762314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:10.762281    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:11.275319 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:11.275282    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:11.443626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:11.443583    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:23:11.443907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:11.443882    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:23:11.460028 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:11.459998    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:12.222291 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:12.222244    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:23:12.222970 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:12.222925    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy" containerID="cri-o://f28d497924927a80dbbe2a34578b18ba4b56383f9d4a5c5f1dc3d078feff5760" gracePeriod=30
Nov 03 12:23:12.775219 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:12.775186    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:13.559512 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:13.556136    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:13.987357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:13.985253    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:23:14.077557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.074471    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:14.105073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105035    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105094    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105218    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105254    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105365    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105463    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.105501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105370    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:14.105674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105628    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:14.105855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.105811    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:14.106073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.106046    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.106203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.106138    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:14.106203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.106147    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.106203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.106183    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\") pod \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\" (UID: \"0ba5f4d9-ecab-43f8-93c4-78072c3912fe\") "
Nov 03 12:23:14.106907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.106871    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:14.107334 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.107311    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.107467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.107339    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.107467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.107354    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.107467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.107368    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.107665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.107480    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.112522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.112489    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:14.116905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.116877    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:14.117365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.116880    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:14.117365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.116905    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx" pod "0ba5f4d9-ecab-43f8-93c4-78072c3912fe" (UID: "0ba5f4d9-ecab-43f8-93c4-78072c3912fe"). InnerVolumeSpecName "kube-api-access-7zbvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:14.237703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.237589    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.237703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.237637    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.237703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.237655    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbvx\" (UniqueName: \"kubernetes.io/projected/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-kube-api-access-7zbvx\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.237703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.237674    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ba5f4d9-ecab-43f8-93c4-78072c3912fe-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:14.273118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.273076    5784 generic.go:356] "Generic (PLEG): container finished" podID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerID="56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799" exitCode=0
Nov 03 12:23:14.273328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.273138    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" event={"ID":"0ba5f4d9-ecab-43f8-93c4-78072c3912fe","Type":"ContainerDied","Data":"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"}
Nov 03 12:23:14.273328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.273189    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5" event={"ID":"0ba5f4d9-ecab-43f8-93c4-78072c3912fe","Type":"ContainerDied","Data":"4be0a96074ed301097d347361c78c89be695c5c525778b6a446137df0ea58a6a"}
Nov 03 12:23:14.273328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.273217    5784 scope.go:122] "RemoveContainer" containerID="56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"
Nov 03 12:23:14.273572 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.273353    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"
Nov 03 12:23:14.318969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.317768    5784 scope.go:122] "RemoveContainer" containerID="56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"
Nov 03 12:23:14.318969 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:23:14.318200    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799\": container with ID starting with 56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799 not found: ID does not exist" containerID="56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"
Nov 03 12:23:14.318969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.318238    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799"} err="failed to get container status \"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799\": rpc error: code = NotFound desc = could not find container \"56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799\": container with ID starting with 56b6ee64e7244ddc6cf3c1d06a70552c19d7ec87ec89b04859986270f9668799 not found: ID does not exist"
Nov 03 12:23:14.433065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.432987    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:23:14.438904 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:14.438864    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-9xbg5"]
Nov 03 12:23:15.472432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:15.471813    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:15.688401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:15.688283    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" path="/var/lib/kubelet/pods/0ba5f4d9-ecab-43f8-93c4-78072c3912fe/volumes"
Nov 03 12:23:16.035640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:16.013326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:16.188081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:16.187497    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k6g4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:23:16.188081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:16.187566    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:16.229616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:16.229482    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-6584f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:23:16.229616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:16.229550    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:23:17.367536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.367502    5784 generic.go:356] "Generic (PLEG): container finished" podID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerID="f28d497924927a80dbbe2a34578b18ba4b56383f9d4a5c5f1dc3d078feff5760" exitCode=0
Nov 03 12:23:17.367944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.367577    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" event={"ID":"788a7a8c-6712-4e58-86c3-29c577a167d7","Type":"ContainerDied","Data":"f28d497924927a80dbbe2a34578b18ba4b56383f9d4a5c5f1dc3d078feff5760"}
Nov 03 12:23:17.519157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.519128    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:23:17.671802 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.671763    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.671999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.671878    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.671999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.671935    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672020    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672063    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672115    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672104    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.672269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672170    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672188    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.672269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672205    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672255    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\") pod \"788a7a8c-6712-4e58-86c3-29c577a167d7\" (UID: \"788a7a8c-6712-4e58-86c3-29c577a167d7\") "
Nov 03 12:23:17.672585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672428    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.672585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672530    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:23:17.672997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672851    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/788a7a8c-6712-4e58-86c3-29c577a167d7-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.672997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672879    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.672997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672893    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.672997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672906    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.672997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.672948    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.689137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.688029    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:23:17.689137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.688342    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.689685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.689262    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:23:17.689685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.689407    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k" pod "788a7a8c-6712-4e58-86c3-29c577a167d7" (UID: "788a7a8c-6712-4e58-86c3-29c577a167d7"). InnerVolumeSpecName "kube-api-access-xhw4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:23:17.774048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.773968    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.774048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.773997    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhw4k\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-kube-api-access-xhw4k\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.774048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.774007    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.774048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.774017    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:17.774048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:17.774026    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/788a7a8c-6712-4e58-86c3-29c577a167d7-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:23:18.375268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.375237    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"
Nov 03 12:23:18.375268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.375247    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs" event={"ID":"788a7a8c-6712-4e58-86c3-29c577a167d7","Type":"ContainerDied","Data":"290c46ee2b497d6ccbcd7cb48095b01290404f7caf6ae4813f0c0b2a80ee56f1"}
Nov 03 12:23:18.375798 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.375316    5784 scope.go:122] "RemoveContainer" containerID="f28d497924927a80dbbe2a34578b18ba4b56383f9d4a5c5f1dc3d078feff5760"
Nov 03 12:23:18.577175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.577144    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:23:18.581927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.581895    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-qdkxs"]
Nov 03 12:23:18.802846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:18.802814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:19.160807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:19.160772    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:19.323551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:19.323514    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:19.479187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:19.479099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:19.682595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:19.682560    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" path="/var/lib/kubelet/pods/788a7a8c-6712-4e58-86c3-29c577a167d7/volumes"
Nov 03 12:23:22.188279 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:22.188238    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:22.353933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:22.353899    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:23.669188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:23.669153    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:26.473310 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:23:26.473272    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:23:28.100749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:28.100711    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:32.711346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:32.711309    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:37.117145 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:37.117106    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:37.718835 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:37.718804    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:37.970224 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:37.970134    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:38.728497 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:38.728461    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:42.261345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:42.261307    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:42.434473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:42.434431    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:42.666990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:42.666951    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:42.824912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:42.824873    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:43.222570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:43.222538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:43.378824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:43.378785    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:44.667467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:44.667431    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:45.302626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:45.302596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:45.460643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:45.460606    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:49.192726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:49.192687    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:49.346779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:49.346744    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:49.604316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:49.604232    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:50.603650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:50.603615    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:51.126348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:51.126313    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:51.287904 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:51.287666    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:51.674250 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:51.674209    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:51.888493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:51.888459    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:53.083203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:53.083170    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:53.409967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:53.409935    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:53.571831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:53.571749    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:55.929066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:55.929030    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:56.477431 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:23:56.477393    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:23:57.839699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:57.839662    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:23:59.565574 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:23:59.565530    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:00.548968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:00.548931    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:05.447437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:05.447400    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:11.514629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:11.514592    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:11.673340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:11.673261    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:11.691839 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:11.691800    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:24:11.692233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:11.692158    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" containerID="cri-o://13553891f7cff89be04a69664e4f652b6f37cd173ed6b9923c7a5567793813d9" gracePeriod=30
Nov 03 12:24:11.842068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:11.841969    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:13.187530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:13.187491    5784 patch_prober.go:29] interesting pod/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:24:13.187922 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:13.187579    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:24:15.989668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:15.989632    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:16.872222 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:16.872191    5784 generic.go:356] "Generic (PLEG): container finished" podID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerID="13553891f7cff89be04a69664e4f652b6f37cd173ed6b9923c7a5567793813d9" exitCode=0
Nov 03 12:24:16.872513 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:16.872289    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" event={"ID":"586bf1ec-f2ea-47a9-861e-34e713b032d3","Type":"ContainerDied","Data":"13553891f7cff89be04a69664e4f652b6f37cd173ed6b9923c7a5567793813d9"}
Nov 03 12:24:16.972332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:16.972274    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:24:17.054071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054040    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054082    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054166    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054228    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054275    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054335    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054451    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054473    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054540    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054568    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\") pod \"586bf1ec-f2ea-47a9-861e-34e713b032d3\" (UID: \"586bf1ec-f2ea-47a9-861e-34e713b032d3\") "
Nov 03 12:24:17.054628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054612    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054746    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054868    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.054964    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.055123    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.055145    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.055160    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/586bf1ec-f2ea-47a9-861e-34e713b032d3-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.055174    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.055200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.055189    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.056595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.056567    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "kube-api-access-8skl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:17.056837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.056815    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:17.057136 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.057113    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:24:17.057198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.057175    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo" pod "586bf1ec-f2ea-47a9-861e-34e713b032d3" (UID: "586bf1ec-f2ea-47a9-861e-34e713b032d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:24:17.156031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.155992    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.156031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.156025    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8skl4\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-kube-api-access-8skl4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.156031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.156038    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.156342 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.156048    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/586bf1ec-f2ea-47a9-861e-34e713b032d3-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:17.880710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.880676    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg" event={"ID":"586bf1ec-f2ea-47a9-861e-34e713b032d3","Type":"ContainerDied","Data":"cbab09188a1ba2b0b8387800cf61d3de781d1cadc93e394286458c009910981f"}
Nov 03 12:24:17.880913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.880718    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"
Nov 03 12:24:17.880913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:17.880735    5784 scope.go:122] "RemoveContainer" containerID="13553891f7cff89be04a69664e4f652b6f37cd173ed6b9923c7a5567793813d9"
Nov 03 12:24:18.081815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:18.081730    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:24:18.127504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:18.127472    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-nlcqg"]
Nov 03 12:24:19.683119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:19.683081    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" path="/var/lib/kubelet/pods/586bf1ec-f2ea-47a9-861e-34e713b032d3/volumes"
Nov 03 12:24:20.040820 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:20.040727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:20.198192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:20.198154    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:20.421514 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:20.421469    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:24.832544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:24.832499    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:26.471901 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:24:26.471866    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:24:28.598195 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.598154    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:24:28.598747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.598396    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" podUID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" containerName="maas-api" containerID="cri-o://2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de" gracePeriod=30
Nov 03 12:24:28.724567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.724534    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:28.926059 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.926034    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:24:28.982499 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.982453    5784 generic.go:356] "Generic (PLEG): container finished" podID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" containerID="2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de" exitCode=0
Nov 03 12:24:28.982707 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.982598    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" event={"ID":"5ab47a9e-ce12-4d68-9c30-ebec8846a641","Type":"ContainerDied","Data":"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"}
Nov 03 12:24:28.982707 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.982651    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q" event={"ID":"5ab47a9e-ce12-4d68-9c30-ebec8846a641","Type":"ContainerDied","Data":"2256be0f97ba57142d4d0794b51b440400973a8d30c0a02f9868756e5343ffda"}
Nov 03 12:24:28.982707 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.982679    5784 scope.go:122] "RemoveContainer" containerID="2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"
Nov 03 12:24:28.982881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:28.982835    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"
Nov 03 12:24:29.025188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.024965    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") pod \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") "
Nov 03 12:24:29.025188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.025021    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\") pod \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") "
Nov 03 12:24:29.025188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.025079    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\") pod \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") "
Nov 03 12:24:29.025188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.025159    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\" (UniqueName: \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\") pod \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\" (UID: \"5ab47a9e-ce12-4d68-9c30-ebec8846a641\") "
Nov 03 12:24:29.028283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.026331    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca" pod "5ab47a9e-ce12-4d68-9c30-ebec8846a641" (UID: "5ab47a9e-ce12-4d68-9c30-ebec8846a641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:24:29.028471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.028444    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch" pod "5ab47a9e-ce12-4d68-9c30-ebec8846a641" (UID: "5ab47a9e-ce12-4d68-9c30-ebec8846a641"). InnerVolumeSpecName "kube-api-access-646ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:24:29.039167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.037984    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs" pod "5ab47a9e-ce12-4d68-9c30-ebec8846a641" (UID: "5ab47a9e-ce12-4d68-9c30-ebec8846a641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:29.039167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.038013    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls" pod "5ab47a9e-ce12-4d68-9c30-ebec8846a641" (UID: "5ab47a9e-ce12-4d68-9c30-ebec8846a641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:24:29.048806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.045803    5784 scope.go:122] "RemoveContainer" containerID="2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"
Nov 03 12:24:29.048806 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:24:29.047284    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de\": container with ID starting with 2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de not found: ID does not exist" containerID="2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"
Nov 03 12:24:29.048806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.047326    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de"} err="failed to get container status \"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de\": rpc error: code = NotFound desc = could not find container \"2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de\": container with ID starting with 2a4cbe75fda320d4eabb89a785c4ac2c74be1248ce3fd2fbc0622ae0f4dc43de not found: ID does not exist"
Nov 03 12:24:29.127347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.127314    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.127347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.127345    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ab47a9e-ce12-4d68-9c30-ebec8846a641-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.127581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.127360    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ab47a9e-ce12-4d68-9c30-ebec8846a641-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.127581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.127373    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-646ch\" (UniqueName: \"kubernetes.io/projected/5ab47a9e-ce12-4d68-9c30-ebec8846a641-kube-api-access-646ch\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:24:29.266491 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.266399    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:29.396558 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.396521    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:24:29.400940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.400907    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-deb666-7c9f98b9c7-n4z6q"]
Nov 03 12:24:29.416231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.416203    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:29.685526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:29.684259    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" path="/var/lib/kubelet/pods/5ab47a9e-ce12-4d68-9c30-ebec8846a641/volumes"
Nov 03 12:24:30.259901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:30.259861    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:30.539482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:30.537832    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:30.701728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:30.701691    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:32.107288 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:32.107251    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:32.660853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:32.660821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:32.898455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:32.898419    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:33.050757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:33.050676    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:33.506733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:33.506699    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:34.965193 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.965126    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.965944    5784 cpu_manager.go:414] "containerMap: removing container" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerName="maas-api"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.965965    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerName="maas-api"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.965994    5784 cpu_manager.go:414] "containerMap: removing container" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966000    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966007    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966012    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966020    5784 cpu_manager.go:414] "containerMap: removing container" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy"
Nov 03 12:24:34.966014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966025    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966032    5784 cpu_manager.go:414] "containerMap: removing container" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966036    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966049    5784 cpu_manager.go:414] "containerMap: removing container" podUID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" containerName="maas-api"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966053    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" containerName="maas-api"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966062    5784 cpu_manager.go:414] "containerMap: removing container" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966067    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966074    5784 cpu_manager.go:414] "containerMap: removing container" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966079    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966155    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="788a7a8c-6712-4e58-86c3-29c577a167d7" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966164    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba5f4d9-ecab-43f8-93c4-78072c3912fe" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966172    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab47a9e-ce12-4d68-9c30-ebec8846a641" containerName="maas-api"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966179    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="404cf2d0-b1cb-423a-b79d-7b19819514b4" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966186    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="586bf1ec-f2ea-47a9-861e-34e713b032d3" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966194    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ba45b6-b88f-4dfb-80b5-30baef2925ae" containerName="maas-api"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966202    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="228facfc-1620-410f-b6f3-51aed832c6ab" containerName="istio-proxy"
Nov 03 12:24:34.966480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.966208    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d81f525b-a10f-4025-80d2-56959593edd1" containerName="istio-proxy"
Nov 03 12:24:34.971071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:34.971053    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.015512 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.015483    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.015512 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.015518    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8nv\" (UniqueName: \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.015732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.015688    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.015767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.015731    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.065567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.065533    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d25504-serving-cert\""
Nov 03 12:24:35.069157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.069125    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:24:35.117004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.116955    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.117004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.116999    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8nv\" (UniqueName: \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.117287 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:24:35.117115    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-serving-cert: secret "maas-api-e2e-ipp-b-d25504-serving-cert" not found
Nov 03 12:24:35.117287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.117152    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.117287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.117203    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.117554 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:24:35.117504    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls podName:ed620fcf-e6e7-4848-9631-8cfe6c639307 nodeName:}" failed. No retries permitted until 2026-11-03 12:24:35.617366273 +0000 UTC m=+1982.059568557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls") pod "maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" (UID: "ed620fcf-e6e7-4848-9631-8cfe6c639307") : secret "maas-api-e2e-ipp-b-d25504-serving-cert" not found
Nov 03 12:24:35.118454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.118427    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.120203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.120183    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.175421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.175373    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8nv\" (UniqueName: \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.199599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.199552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:35.355552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.355466    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:35.514596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.514561    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:35.623112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.623026    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.625465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.625440    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:35.884198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:35.884170    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:36.164683 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:36.164604    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:36.257504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:36.257472    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:24:36.259754 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:24:36.259728    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded620fcf_e6e7_4848_9631_8cfe6c639307.slice/crio-1ffda19072a3a3e5c86dc46b7d1fa4f0cf694a47513423eb232d9b83a4ffefce WatchSource:0}: Error finding container 1ffda19072a3a3e5c86dc46b7d1fa4f0cf694a47513423eb232d9b83a4ffefce: Status 404 returned error can't find the container with id 1ffda19072a3a3e5c86dc46b7d1fa4f0cf694a47513423eb232d9b83a4ffefce
Nov 03 12:24:37.056209 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:37.056174    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" event={"ID":"ed620fcf-e6e7-4848-9631-8cfe6c639307","Type":"ContainerStarted","Data":"1ffda19072a3a3e5c86dc46b7d1fa4f0cf694a47513423eb232d9b83a4ffefce"}
Nov 03 12:24:38.094924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:38.094891    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" event={"ID":"ed620fcf-e6e7-4848-9631-8cfe6c639307","Type":"ContainerStarted","Data":"d4310fa365511e2599bdff00fe1bbb485b863404f1e155ec5d1fda46a124f1b3"}
Nov 03 12:24:38.095340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:38.094971    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:38.203217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:38.203153    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" podStartSLOduration=3.282835506 podStartE2EDuration="4.203136431s" totalImagesPullingTime="920.300925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:24:38.199611802 +0000 UTC m=+1984.641814147" watchObservedRunningTime="2026-11-03 12:24:38.203136431 +0000 UTC m=+1984.645338737"
Nov 03 12:24:39.360186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:39.360155    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:39.524732 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:39.524686    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:40.674566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:40.674527    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:41.220628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:41.220591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:41.379903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:41.379870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:42.472083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:42.472053    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:44.110550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:44.110521    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:24:44.861634 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:44.861584    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:45.015768 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:45.015733    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:46.252537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:46.252509    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:46.800939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:46.800909    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:46.966476 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:46.966413    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:47.265445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:47.265407    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:48.277773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:48.277736    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:48.826741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:48.826682    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:49.544986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:49.537973    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:50.301401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:50.298019    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:50.459801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:50.459713    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:50.653401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:50.653347    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:51.435448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:51.435373    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:51.585426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:51.585369    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:51.833641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:51.833548    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:52.315661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:52.315623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:55.048887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:55.048850    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:56.217179 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:56.217144    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:56.472803 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:24:56.472705    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:24:58.204125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:58.204089    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:58.390158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:58.390102    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:24:58.693398 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:24:58.693007    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:02.527789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:02.527752    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:02.789120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:02.789024    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:03.354237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:03.354202    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:06.405698 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:06.405670    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:07.590775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:07.590736    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:08.046093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:08.046058    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:08.200236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:08.200194    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:08.367906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:08.367819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:12.877073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:12.876862    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:13.047730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:13.047689    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:13.218409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:13.218282    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:15.974537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:15.974499    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:16.145001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:16.144967    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:17.717644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:17.717596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:22.080244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:22.080209    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:26.472096 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:26.472061    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:25:26.491433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:26.491401    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:27.784131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:27.784098    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:25:27.784723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:27.784306    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" containerID="cri-o://5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27" gracePeriod=30
Nov 03 12:25:29.994086 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:29.994045    5784 patch_prober.go:29] interesting pod/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:25:29.994503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:29.994126    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:25:31.124194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:31.124158    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:33.047723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.047693    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:25:33.174678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174648    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.174874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174734    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.174874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174781    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.174874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174813    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.174874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174829    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.175113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174883    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.175113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174925    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.175113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.174993    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.175113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175060    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:33.175113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175094    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\") pod \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\" (UID: \"f53c6021-1b1d-4c36-bec0-07bafc6c5446\") "
Nov 03 12:25:33.175418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175295    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:33.175486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175416    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:25:33.175560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175538    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:33.175631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175226    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:33.175870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175849    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.175953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175877    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.175953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175894    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.175953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175910    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.175953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.175925    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.177132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.177110    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "kube-api-access-b8zhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:33.177238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.177148    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:25:33.177294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.177101    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:33.186568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.177303    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token" pod "f53c6021-1b1d-4c36-bec0-07bafc6c5446" (UID: "f53c6021-1b1d-4c36-bec0-07bafc6c5446"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:33.276693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.276656    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8zhb\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-kube-api-access-b8zhb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.276693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.276686    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.276693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.276698    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.276940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.276710    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f53c6021-1b1d-4c36-bec0-07bafc6c5446-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:33.658686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.658651    5784 generic.go:356] "Generic (PLEG): container finished" podID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerID="5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27" exitCode=0
Nov 03 12:25:33.658891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.658738    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"
Nov 03 12:25:33.658891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.658733    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" event={"ID":"f53c6021-1b1d-4c36-bec0-07bafc6c5446","Type":"ContainerDied","Data":"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"}
Nov 03 12:25:33.658891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.658884    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw" event={"ID":"f53c6021-1b1d-4c36-bec0-07bafc6c5446","Type":"ContainerDied","Data":"a2d1e4b622f9540676d6d24aa4f9df5e3ef50d10ee44eb3aab2ba2db177e7ab0"}
Nov 03 12:25:33.659072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.658907    5784 scope.go:122] "RemoveContainer" containerID="5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"
Nov 03 12:25:33.669981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.669950    5784 scope.go:122] "RemoveContainer" containerID="5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"
Nov 03 12:25:33.670273 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:33.670249    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27\": container with ID starting with 5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27 not found: ID does not exist" containerID="5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"
Nov 03 12:25:33.670352 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.670280    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27"} err="failed to get container status \"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27\": rpc error: code = NotFound desc = could not find container \"5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27\": container with ID starting with 5a000b0285d7eb2893bfc3b6a184ff49ab60b7b93b71529cf2664e8b0939cd27 not found: ID does not exist"
Nov 03 12:25:33.812395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.812335    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:25:33.815613 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:33.815586    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-g2cgw"]
Nov 03 12:25:35.680915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:35.680880    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" path="/var/lib/kubelet/pods/f53c6021-1b1d-4c36-bec0-07bafc6c5446/volumes"
Nov 03 12:25:35.728960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:35.728929    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:39.476241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:39.476207    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:25:39.476755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:39.476490    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" containerID="cri-o://446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3" gracePeriod=30
Nov 03 12:25:39.730996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:39.730890    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:39.909101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:39.909061    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:40.066773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:40.066674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:41.680836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:41.680785    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:41.828553 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:41.828517    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:42.016286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:42.016196    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:44.293266 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.293213    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:25:44.293734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.293296    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:25:44.741519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.741495    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:25:44.759202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.759169    5784 generic.go:356] "Generic (PLEG): container finished" podID="2a232601-351d-41f5-9be2-86b807416807" containerID="446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3" exitCode=0
Nov 03 12:25:44.759363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.759212    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"
Nov 03 12:25:44.759363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.759250    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" event={"ID":"2a232601-351d-41f5-9be2-86b807416807","Type":"ContainerDied","Data":"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"}
Nov 03 12:25:44.759363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.759314    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk" event={"ID":"2a232601-351d-41f5-9be2-86b807416807","Type":"ContainerDied","Data":"5f7d03c6b732b418f74027950e5940c9e980118095190f87a93fb9b2dbbf8efd"}
Nov 03 12:25:44.759363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.759343    5784 scope.go:122] "RemoveContainer" containerID="446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"
Nov 03 12:25:44.769081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.769064    5784 scope.go:122] "RemoveContainer" containerID="446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"
Nov 03 12:25:44.769362 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:44.769343    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3\": container with ID starting with 446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3 not found: ID does not exist" containerID="446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"
Nov 03 12:25:44.769429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.769372    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3"} err="failed to get container status \"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3\": rpc error: code = NotFound desc = could not find container \"446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3\": container with ID starting with 446f5c65f4e3ea865cefc0181481831313cb6a3519a4bd3c684db047a24f60e3 not found: ID does not exist"
Nov 03 12:25:44.804814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.804787    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.804945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.804851    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.804986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.804962    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805012    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805059 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805052    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805087    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805170 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805087    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:44.805227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805168    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805210    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805296    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:44.805434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805320    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:44.805434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805349    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\") pod \"2a232601-351d-41f5-9be2-86b807416807\" (UID: \"2a232601-351d-41f5-9be2-86b807416807\") "
Nov 03 12:25:44.805538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805463    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:25:44.805591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805571    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:44.805925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805904    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a232601-351d-41f5-9be2-86b807416807-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.806032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805930    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.806032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805945    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.806032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805962    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.806032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.805976    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.807107 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.807084    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "kube-api-access-dz67b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:44.807541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.807521    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:25:44.807636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.807537    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:25:44.807674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.807629    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token" pod "2a232601-351d-41f5-9be2-86b807416807" (UID: "2a232601-351d-41f5-9be2-86b807416807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:25:44.908330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.908291    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz67b\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-kube-api-access-dz67b\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.908330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.908330    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a232601-351d-41f5-9be2-86b807416807-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.908330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.908346    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a232601-351d-41f5-9be2-86b807416807-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:44.908689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:44.908369    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a232601-351d-41f5-9be2-86b807416807-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:25:45.173586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.173506    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:25:45.178145 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.178119    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-m26mk"]
Nov 03 12:25:45.681612 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.681577    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a232601-351d-41f5-9be2-86b807416807" path="/var/lib/kubelet/pods/2a232601-351d-41f5-9be2-86b807416807/volumes"
Nov 03 12:25:45.962567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.962468    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:25:45.963310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963289    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy"
Nov 03 12:25:45.963310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963309    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy"
Nov 03 12:25:45.963479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963321    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy"
Nov 03 12:25:45.963479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963327    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy"
Nov 03 12:25:45.963479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963421    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a232601-351d-41f5-9be2-86b807416807" containerName="istio-proxy"
Nov 03 12:25:45.963479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.963431    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53c6021-1b1d-4c36-bec0-07bafc6c5446" containerName="istio-proxy"
Nov 03 12:25:45.968214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:45.968193    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.016618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.016569    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-607dc3-serving-cert\""
Nov 03 12:25:46.020296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.020265    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.020588 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.020529    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.020703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.020668    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.020918 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.020897    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfnmh\" (UniqueName: \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.065969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.065934    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:25:46.122061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.122023    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfnmh\" (UniqueName: \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.122061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.122066    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.122324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.122094    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.122324 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.122140    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.122995 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:46.122477    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-serving-cert: secret "maas-api-e2e-praxis-a-607dc3-serving-cert" not found
Nov 03 12:25:46.122995 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:46.122569    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls podName:f187e8ec-c241-4d73-ba45-84f7a4843da3 nodeName:}" failed. No retries permitted until 2026-11-03 12:25:46.622542297 +0000 UTC m=+2053.064744599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls") pod "maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" (UID: "f187e8ec-c241-4d73-ba45-84f7a4843da3") : secret "maas-api-e2e-praxis-a-607dc3-serving-cert" not found
Nov 03 12:25:46.123259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.123234    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.125345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.125323    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.172731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.172691    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:46.174557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.174539    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfnmh\" (UniqueName: \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.323236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.323153    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:46.478046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.478008    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:46.628343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.628312    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.630905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.630876    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") pod \"maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:46.883215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:46.883136    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:47.254014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:47.253958    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:25:47.255538 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:25:47.255513    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf187e8ec_c241_4d73_ba45_84f7a4843da3.slice/crio-2140f27cd0ab7d123de18aa471197b3f3f7be2ebfd7fb7fc3f27a9147e26b3e2 WatchSource:0}: Error finding container 2140f27cd0ab7d123de18aa471197b3f3f7be2ebfd7fb7fc3f27a9147e26b3e2: Status 404 returned error can't find the container with id 2140f27cd0ab7d123de18aa471197b3f3f7be2ebfd7fb7fc3f27a9147e26b3e2
Nov 03 12:25:47.257673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:47.257655    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:25:47.789226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:47.789188    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" event={"ID":"f187e8ec-c241-4d73-ba45-84f7a4843da3","Type":"ContainerStarted","Data":"2140f27cd0ab7d123de18aa471197b3f3f7be2ebfd7fb7fc3f27a9147e26b3e2"}
Nov 03 12:25:48.806359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:48.806325    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" event={"ID":"f187e8ec-c241-4d73-ba45-84f7a4843da3","Type":"ContainerStarted","Data":"7735cdeaf0e88129c7e3c1db20d7ea172bc22746df32f6d727ac1b51f02ca54f"}
Nov 03 12:25:48.807221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:48.807202    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:48.913675 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:48.913615    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" podStartSLOduration=2.850453485 podStartE2EDuration="3.913599061s" totalImagesPullingTime="1.063145576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:25:48.912242578 +0000 UTC m=+2055.354444884" watchObservedRunningTime="2026-11-03 12:25:48.913599061 +0000 UTC m=+2055.355801365"
Nov 03 12:25:50.814196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:50.814163    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:51.932452 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:51.932425    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:55.226102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:55.226054    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:25:55.826750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:55.826715    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:25:56.472153 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:25:56.472119    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:25:59.709510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:25:59.709477    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:01.273479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:01.273448    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:01.913792 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:01.913741    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:02.328610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:02.328496    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:03.862509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.862471    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:26:03.867431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.867407    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:03.918294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.918262    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:26:03.937807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.937777    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:03.938002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.937813    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:03.938002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.937937    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d7qh\" (UniqueName: \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:03.938002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.937967    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:03.960069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:03.960028    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-980a31-serving-cert\""
Nov 03 12:26:04.039106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.039064    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.039326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.039118    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.039326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.039246    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d7qh\" (UniqueName: \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.039326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.039275    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.039546 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:26:04.039242    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-serving-cert: secret "maas-api-e2e-praxis-b-980a31-serving-cert" not found
Nov 03 12:26:04.039895 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:26:04.039876    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls podName:9289d555-8aed-4467-aa19-e1a1fb15c348 nodeName:}" failed. No retries permitted until 2026-11-03 12:26:04.539851202 +0000 UTC m=+2070.982053485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls") pod "maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" (UID: "9289d555-8aed-4467-aa19-e1a1fb15c348") : secret "maas-api-e2e-praxis-b-980a31-serving-cert" not found
Nov 03 12:26:04.040054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.039963    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.042011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.041985    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.074232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.074196    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:04.095363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.095330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d7qh\" (UniqueName: \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.241457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.241419    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:04.396408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.396343    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:04.545431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.545321    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.547931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.547904    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") pod \"maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:04.782885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:04.782854    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:05.162527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:05.162035    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:26:05.163736 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:26:05.163706    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9289d555_8aed_4467_aa19_e1a1fb15c348.slice/crio-78aac0db5fb0f766e1106ed18ee0e7fd6c6b5ee97b4e7a7b148fd140e38d1a44 WatchSource:0}: Error finding container 78aac0db5fb0f766e1106ed18ee0e7fd6c6b5ee97b4e7a7b148fd140e38d1a44: Status 404 returned error can't find the container with id 78aac0db5fb0f766e1106ed18ee0e7fd6c6b5ee97b4e7a7b148fd140e38d1a44
Nov 03 12:26:05.959340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:05.959303    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" event={"ID":"9289d555-8aed-4467-aa19-e1a1fb15c348","Type":"ContainerStarted","Data":"78aac0db5fb0f766e1106ed18ee0e7fd6c6b5ee97b4e7a7b148fd140e38d1a44"}
Nov 03 12:26:06.494600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:06.494564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:06.658464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:06.658423    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:06.813210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:06.813128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:06.970359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:06.970322    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" event={"ID":"9289d555-8aed-4467-aa19-e1a1fb15c348","Type":"ContainerStarted","Data":"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"}
Nov 03 12:26:06.970519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:06.970430    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:07.077913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:07.077789    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" podStartSLOduration=3.222207722 podStartE2EDuration="4.077767832s" totalImagesPullingTime="855.56011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:26:07.076453226 +0000 UTC m=+2073.518655552" watchObservedRunningTime="2026-11-03 12:26:07.077767832 +0000 UTC m=+2073.519970138"
Nov 03 12:26:09.058637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:09.058597    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:09.588087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:09.588053    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:09.761307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:09.761276    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:12.983409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:12.983319    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:26:14.006786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:14.006756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:19.097683 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:19.097650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:22.546687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:22.546650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:23.028490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:23.028458    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:25.624785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:25.624754    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:26.472404 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:26:26.472349    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:26:30.494539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:30.494499    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:34.075349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:34.075318    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:26:34.084178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:34.084155    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:26:35.110951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:35.110921    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:36.347164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:36.347102    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:41.017734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:41.017702    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:45.615564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:45.615516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:47.427797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:47.427764    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:48.628188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:48.628137    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:49.241453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:49.237242    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:49.571232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:49.571141    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:50.256010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:50.255947    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:50.451493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:50.451455    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:52.321887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:52.321845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:52.473286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:52.473259    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:53.542645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:53.542607    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:53.691078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:53.691037    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:53.856856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:53.856776    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:54.780238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:54.780197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:55.209879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:55.209818    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:55.688767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:55.688708    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:55.915505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:55.915469    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:56.471871 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:26:56.471830    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:26:56.715554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:56.715523    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:56.875570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:56.875475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:26:59.976061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:26:59.976029    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:00.157409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:00.157344    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:00.321850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:00.321771    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:03.059146 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:03.059111    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:03.242435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:03.242365    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:04.749187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:04.749145    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:09.165322 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.165275    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:27:09.169991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.169970    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.219452 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.219426    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c02ae4-serving-cert\""
Nov 03 12:27:09.269394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.269351    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:27:09.282466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.282439    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.282633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.282522    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.282679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.282646    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdtr6\" (UniqueName: \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.282760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.282745    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.380836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.380798    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:09.384111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.384083    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.384242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.384146    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.384242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.384174    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.384242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.384232    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdtr6\" (UniqueName: \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.384431 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:09.384312    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-serving-cert: secret "maas-api-e2e-ipp-a-c02ae4-serving-cert" not found
Nov 03 12:27:09.384431 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:09.384404    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls podName:622b8886-e59b-4bb2-9e6c-d83372b89eb0 nodeName:}" failed. No retries permitted until 2026-11-03 12:27:09.884368896 +0000 UTC m=+2136.326571178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls") pod "maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" (UID: "622b8886-e59b-4bb2-9e6c-d83372b89eb0") : secret "maas-api-e2e-ipp-a-c02ae4-serving-cert" not found
Nov 03 12:27:09.384871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.384851    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.386687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.386667    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.442373 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.442287    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdtr6\" (UniqueName: \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.536700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.536672    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:09.701464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.701389    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:09.889786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.889753    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:09.892142 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:09.892122    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:10.084649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:10.084559    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:10.195741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:10.195576    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:10.457535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:10.457503    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:27:10.458815 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:27:10.458783    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod622b8886_e59b_4bb2_9e6c_d83372b89eb0.slice/crio-55586e38d266231ba5779c4ec4994145011e2f050dbbcc0526505d3b530581f2 WatchSource:0}: Error finding container 55586e38d266231ba5779c4ec4994145011e2f050dbbcc0526505d3b530581f2: Status 404 returned error can't find the container with id 55586e38d266231ba5779c4ec4994145011e2f050dbbcc0526505d3b530581f2
Nov 03 12:27:11.464456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:11.464416    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" event={"ID":"622b8886-e59b-4bb2-9e6c-d83372b89eb0","Type":"ContainerStarted","Data":"55586e38d266231ba5779c4ec4994145011e2f050dbbcc0526505d3b530581f2"}
Nov 03 12:27:12.477901 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:12.477857    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" event={"ID":"622b8886-e59b-4bb2-9e6c-d83372b89eb0","Type":"ContainerStarted","Data":"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"}
Nov 03 12:27:12.478404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:12.477937    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:12.586843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:12.586755    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" podStartSLOduration=2.539983726 podStartE2EDuration="3.586739376s" totalImagesPullingTime="1.04675565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:27:12.583418453 +0000 UTC m=+2139.025620779" watchObservedRunningTime="2026-11-03 12:27:12.586739376 +0000 UTC m=+2139.028941681"
Nov 03 12:27:13.830633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:13.830593    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:14.314919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:14.314885    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:18.490845 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:18.490815    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:27:18.727267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:18.727214    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:19.413420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:19.413388    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:19.570643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:19.570586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:19.824539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:19.824453    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:23.629773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:23.629731    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:25.029668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:25.029629    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:27:25.030127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:25.029822    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" containerID="cri-o://627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6" gracePeriod=30
Nov 03 12:27:25.690249 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:25.690206    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:26.472498 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:26.472461    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:27:28.601358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:28.601325    5784 patch_prober.go:29] interesting pod/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:27:28.601751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:28.601402    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:27:29.623666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:29.623631    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:30.299729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.299705    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:27:30.350185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350097    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350166    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350287    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350431    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350451    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:30.350600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350491    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350553    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350585    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350613    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350639    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\") pod \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\" (UID: \"9f09b10e-f664-43bc-ab2c-ba3fa691dea9\") "
Nov 03 12:27:30.350771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.350748    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:30.351058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351031    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:27:30.351416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351184    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.351416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351206    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.351416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351220    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.351416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351308    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:30.351416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.351358    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:30.352960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.352789    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "kube-api-access-6l8bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:30.352960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.352843    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:30.353172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.353149    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:30.353463 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.353438    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo" pod "9f09b10e-f664-43bc-ab2c-ba3fa691dea9" (UID: "9f09b10e-f664-43bc-ab2c-ba3fa691dea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:27:30.452822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452784    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l8bj\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-kube-api-access-6l8bj\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.452822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452821    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.453021 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452838    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.453021 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452853    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.453021 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452867    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.453021 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.452880    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09b10e-f664-43bc-ab2c-ba3fa691dea9-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:30.602011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.601924    5784 generic.go:356] "Generic (PLEG): container finished" podID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerID="627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6" exitCode=0
Nov 03 12:27:30.602011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.601972    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"
Nov 03 12:27:30.602255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.602009    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" event={"ID":"9f09b10e-f664-43bc-ab2c-ba3fa691dea9","Type":"ContainerDied","Data":"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"}
Nov 03 12:27:30.602255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.602095    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6" event={"ID":"9f09b10e-f664-43bc-ab2c-ba3fa691dea9","Type":"ContainerDied","Data":"0552cea753971ae53da7df1cb9eed210b343e6fc2b8c9da1645799eae78979d0"}
Nov 03 12:27:30.602255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.602127    5784 scope.go:122] "RemoveContainer" containerID="627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"
Nov 03 12:27:30.614107 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.614087    5784 scope.go:122] "RemoveContainer" containerID="627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"
Nov 03 12:27:30.614413 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:30.614363    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6\": container with ID starting with 627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6 not found: ID does not exist" containerID="627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"
Nov 03 12:27:30.614488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.614427    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6"} err="failed to get container status \"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6\": rpc error: code = NotFound desc = could not find container \"627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6\": container with ID starting with 627214d100b8de0699cee60215896952a728dcb48d068c0c0a8272f5ba0acfb6 not found: ID does not exist"
Nov 03 12:27:30.756206 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.756169    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:27:30.760065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:30.760042    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5951bc-openshift-default-5c5dcb445-rpwf6"]
Nov 03 12:27:31.682338 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:31.682304    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" path="/var/lib/kubelet/pods/9f09b10e-f664-43bc-ab2c-ba3fa691dea9/volumes"
Nov 03 12:27:34.013023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:34.012989    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:38.174563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:38.174527    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:38.322159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:38.322128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:38.549780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:38.549689    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:42.274821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:42.274733    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:27:42.275432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:42.275000    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" containerID="cri-o://1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518" gracePeriod=30
Nov 03 12:27:44.538753 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:44.538716    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:45.310999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:45.310962    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:27:45.311182 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:45.311035    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:27:47.548124 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.548045    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:27:47.655565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655532    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.655565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655581    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.655832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655701    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.655832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655732    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.655832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655759    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.656050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655883    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.656050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.655884    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:47.656151 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656132    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:47.656277 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656257    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.656365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656310    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.656365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656333    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:27:47.656917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656661    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:47.656917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656663    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\") pod \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\" (UID: \"ae1de832-a360-4f45-beb5-ad9d1a191f9c\") "
Nov 03 12:27:47.657096 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.656918    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:47.657742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.657648    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.657742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.657675    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.657742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.657690    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.657742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.657707    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.657742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.657720    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.659003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.658863    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:47.659161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.659132    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:27:47.659227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.659208    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "kube-api-access-jxlc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:27:47.659416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.659360    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo" pod "ae1de832-a360-4f45-beb5-ad9d1a191f9c" (UID: "ae1de832-a360-4f45-beb5-ad9d1a191f9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:27:47.720649 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.720613    5784 generic.go:356] "Generic (PLEG): container finished" podID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerID="1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518" exitCode=0
Nov 03 12:27:47.720854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.720816    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" event={"ID":"ae1de832-a360-4f45-beb5-ad9d1a191f9c","Type":"ContainerDied","Data":"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"}
Nov 03 12:27:47.720933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.720869    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv" event={"ID":"ae1de832-a360-4f45-beb5-ad9d1a191f9c","Type":"ContainerDied","Data":"3338d23af9913c3c784f005076366c4875c67b77568af6515a4693d9fb86a8c0"}
Nov 03 12:27:47.720933 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.720898    5784 scope.go:122] "RemoveContainer" containerID="1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"
Nov 03 12:27:47.721078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.721062    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"
Nov 03 12:27:47.735455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.735371    5784 scope.go:122] "RemoveContainer" containerID="1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"
Nov 03 12:27:47.735844 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:47.735810    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518\": container with ID starting with 1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518 not found: ID does not exist" containerID="1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"
Nov 03 12:27:47.735936 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.735850    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518"} err="failed to get container status \"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518\": rpc error: code = NotFound desc = could not find container \"1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518\": container with ID starting with 1709a395cd522baabee969dfc91ed423fa5a0802e0ebf0d84c76b7454cf09518 not found: ID does not exist"
Nov 03 12:27:47.758745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.758716    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.758745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.758744    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.758936 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.758759    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxlc4\" (UniqueName: \"kubernetes.io/projected/ae1de832-a360-4f45-beb5-ad9d1a191f9c-kube-api-access-jxlc4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.758936 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.758774    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1de832-a360-4f45-beb5-ad9d1a191f9c-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:27:47.875298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.875210    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:27:47.880048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:47.880022    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-c48tv"]
Nov 03 12:27:48.893588 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:48.893557    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:49.682259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:49.682223    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" path="/var/lib/kubelet/pods/ae1de832-a360-4f45-beb5-ad9d1a191f9c/volumes"
Nov 03 12:27:52.571222 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:52.571175    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:52.732295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:52.732254    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:54.076468 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.076430    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:27:54.077022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.076998    5784 cpu_manager.go:414] "containerMap: removing container" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy"
Nov 03 12:27:54.077022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.077017    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy"
Nov 03 12:27:54.077181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.077064    5784 cpu_manager.go:414] "containerMap: removing container" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy"
Nov 03 12:27:54.077181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.077074    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy"
Nov 03 12:27:54.077253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.077193    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f09b10e-f664-43bc-ab2c-ba3fa691dea9" containerName="istio-proxy"
Nov 03 12:27:54.077253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.077206    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1de832-a360-4f45-beb5-ad9d1a191f9c" containerName="istio-proxy"
Nov 03 12:27:54.082135 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.082115    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.130561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.130516    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.130738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.130703    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzb9z\" (UniqueName: \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.130862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.130831    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.130931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.130900    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.133643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.133617    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:27:54.169700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.169674    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-40ffcd-serving-cert\""
Nov 03 12:27:54.232598 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.232566    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.232782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.232619    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.232782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.232675    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.232782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.232745    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzb9z\" (UniqueName: \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.232961 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:54.232777    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-serving-cert: secret "maas-api-e2e-praxis-b-40ffcd-serving-cert" not found
Nov 03 12:27:54.232961 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:54.232854    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls podName:4aa19ce4-982a-41f0-9f82-5d2c2e973844 nodeName:}" failed. No retries permitted until 2026-11-03 12:27:54.732831461 +0000 UTC m=+2181.175033757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls") pod "maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" (UID: "4aa19ce4-982a-41f0-9f82-5d2c2e973844") : secret "maas-api-e2e-praxis-b-40ffcd-serving-cert" not found
Nov 03 12:27:54.233427 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.233403    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.235007 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.234987    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.285751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.285719    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzb9z\" (UniqueName: \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.323150 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.321860    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:54.480844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.480814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:54.631311 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.631273    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:54.738628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.738532    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.741252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.741224    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") pod \"maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:54.995353 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:54.995268    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:55.360673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:55.360644    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:27:55.362413 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:27:55.362363    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aa19ce4_982a_41f0_9f82_5d2c2e973844.slice/crio-ef83a5e70a6e19b213e4614fd3f7f5b756c83cb062c955ece6e57672ac8b3650 WatchSource:0}: Error finding container ef83a5e70a6e19b213e4614fd3f7f5b756c83cb062c955ece6e57672ac8b3650: Status 404 returned error can't find the container with id ef83a5e70a6e19b213e4614fd3f7f5b756c83cb062c955ece6e57672ac8b3650
Nov 03 12:27:55.785160 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:55.785126    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" event={"ID":"4aa19ce4-982a-41f0-9f82-5d2c2e973844","Type":"ContainerStarted","Data":"ef83a5e70a6e19b213e4614fd3f7f5b756c83cb062c955ece6e57672ac8b3650"}
Nov 03 12:27:56.471413 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:27:56.471297    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:27:56.793531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:56.793433    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" event={"ID":"4aa19ce4-982a-41f0-9f82-5d2c2e973844","Type":"ContainerStarted","Data":"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"}
Nov 03 12:27:56.793531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:56.793501    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:27:56.915590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:56.915530    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" podStartSLOduration=2.125723307 podStartE2EDuration="2.915513499s" totalImagesPullingTime="789.790192ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:27:56.914638531 +0000 UTC m=+2183.356840838" watchObservedRunningTime="2026-11-03 12:27:56.915513499 +0000 UTC m=+2183.357715805"
Nov 03 12:27:57.185794 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:57.185755    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:57.351656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:57.351623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:58.703217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:58.703182    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:59.150210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:59.150165    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:59.303534 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:59.303483    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:27:59.973613 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:27:59.973581    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:02.805364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:02.805332    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:28:03.777998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:03.777964    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:03.930064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:03.930032    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:04.082308 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:04.082229    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:05.633310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:05.633272    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:05.836315 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:05.836280    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:05.996501 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:05.996411    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:08.538645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:08.538591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:11.413797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:11.413744    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:12.800980 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:12.800945    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:28:12.801535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:12.801223    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerName="maas-api" containerID="cri-o://f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a" gracePeriod=30
Nov 03 12:28:12.915530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:12.915499    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:12.977888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:12.977849    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.101:8443/health\": dial tcp 10.134.0.101:8443: connect: connection refused"
Nov 03 12:28:13.098521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.098492    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:28:13.274066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.274021    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") pod \"9289d555-8aed-4467-aa19-e1a1fb15c348\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") "
Nov 03 12:28:13.274361 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.274085    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\") pod \"9289d555-8aed-4467-aa19-e1a1fb15c348\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") "
Nov 03 12:28:13.274361 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.274116    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\") pod \"9289d555-8aed-4467-aa19-e1a1fb15c348\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") "
Nov 03 12:28:13.274361 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.274179    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\" (UniqueName: \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\") pod \"9289d555-8aed-4467-aa19-e1a1fb15c348\" (UID: \"9289d555-8aed-4467-aa19-e1a1fb15c348\") "
Nov 03 12:28:13.274952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.274698    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca" pod "9289d555-8aed-4467-aa19-e1a1fb15c348" (UID: "9289d555-8aed-4467-aa19-e1a1fb15c348"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:28:13.277098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.277065    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls" pod "9289d555-8aed-4467-aa19-e1a1fb15c348" (UID: "9289d555-8aed-4467-aa19-e1a1fb15c348"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:28:13.277218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.277102    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs" pod "9289d555-8aed-4467-aa19-e1a1fb15c348" (UID: "9289d555-8aed-4467-aa19-e1a1fb15c348"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:28:13.278365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.278345    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh" pod "9289d555-8aed-4467-aa19-e1a1fb15c348" (UID: "9289d555-8aed-4467-aa19-e1a1fb15c348"). InnerVolumeSpecName "kube-api-access-8d7qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:28:13.376339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.376307    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d7qh\" (UniqueName: \"kubernetes.io/projected/9289d555-8aed-4467-aa19-e1a1fb15c348-kube-api-access-8d7qh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:13.376552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.376346    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:13.376552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.376360    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9289d555-8aed-4467-aa19-e1a1fb15c348-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:13.376552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.376372    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9289d555-8aed-4467-aa19-e1a1fb15c348-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:28:13.918674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.918632    5784 generic.go:356] "Generic (PLEG): container finished" podID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerID="f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a" exitCode=0
Nov 03 12:28:13.919174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.918696    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"
Nov 03 12:28:13.919174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.918713    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" event={"ID":"9289d555-8aed-4467-aa19-e1a1fb15c348","Type":"ContainerDied","Data":"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"}
Nov 03 12:28:13.919174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.918793    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv" event={"ID":"9289d555-8aed-4467-aa19-e1a1fb15c348","Type":"ContainerDied","Data":"78aac0db5fb0f766e1106ed18ee0e7fd6c6b5ee97b4e7a7b148fd140e38d1a44"}
Nov 03 12:28:13.919174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.918825    5784 scope.go:122] "RemoveContainer" containerID="f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"
Nov 03 12:28:13.931865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.931841    5784 scope.go:122] "RemoveContainer" containerID="f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"
Nov 03 12:28:13.932191 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:28:13.932164    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a\": container with ID starting with f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a not found: ID does not exist" containerID="f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"
Nov 03 12:28:13.932250 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:13.932201    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a"} err="failed to get container status \"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a\": rpc error: code = NotFound desc = could not find container \"f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a\": container with ID starting with f56e8d51c2d9b48189321d81006870d6778b23fb676a07003f6ab1208b23cb0a not found: ID does not exist"
Nov 03 12:28:14.072433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:14.072305    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:28:14.118188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:14.118151    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-980a31-5dc787f45-rtvwv"]
Nov 03 12:28:14.713814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:14.713764    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:15.681693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:15.681656    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" path="/var/lib/kubelet/pods/9289d555-8aed-4467-aa19-e1a1fb15c348/volumes"
Nov 03 12:28:16.894233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:16.894187    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:17.055686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:17.055647    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:18.256627 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:18.256585    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:18.726842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:18.726791    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:19.529695 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:19.529659    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:20.825744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:20.825713    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:21.259587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:21.259556    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:21.422313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:21.422285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:24.306462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:24.306426    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:26.477428 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:28:26.477361    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:28:29.291390 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:29.291339    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:35.524694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:35.524660    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:39.909902 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:39.909871    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:44.545267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:44.545234    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:50.533915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:50.533873    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:52.891736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:52.891700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:54.921499 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:54.921464    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:28:56.472801 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:28:56.472762    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:28:59.326280 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:28:59.326245    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:03.769770 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:03.769740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:06.655575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:06.655530    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:08.416497 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:08.416462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:08.852303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:08.852218    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:10.980939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:10.980909    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:29:10.981757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:10.981728    5784 cpu_manager.go:414] "containerMap: removing container" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerName="maas-api"
Nov 03 12:29:10.981757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:10.981751    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerName="maas-api"
Nov 03 12:29:10.981935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:10.981842    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9289d555-8aed-4467-aa19-e1a1fb15c348" containerName="maas-api"
Nov 03 12:29:10.985478 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:10.985458    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.034506 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.034479    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-91cd99-openshift-default-dockercfg-pj65x\""
Nov 03 12:29:11.084269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.084184    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:29:11.152851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.152810    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.152851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.152852    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlb6\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.152877    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.152998    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.153050    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.153160    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.153279    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.153345    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.153433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.153413    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255041    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255142    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255185    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255216    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255273    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255298    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlb6\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255331    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255414    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255488    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.255870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255847    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.256007 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255978    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.256114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255869    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.256114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.255900    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.256114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.256039    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.257867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.257845    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.258260 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.258233    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.309040 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.309004    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.309227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.309083    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlb6\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:11.599136 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:11.599052    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:12.013640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.013158    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:29:12.014316 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:29:12.014284    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038643c5_8fdf_4b50_bad8_4ee059945d89.slice/crio-1e02cc3361ed5f3df5987c88cd23468ea40180e75db38e1076946dba1bd4e59f WatchSource:0}: Error finding container 1e02cc3361ed5f3df5987c88cd23468ea40180e75db38e1076946dba1bd4e59f: Status 404 returned error can't find the container with id 1e02cc3361ed5f3df5987c88cd23468ea40180e75db38e1076946dba1bd4e59f
Nov 03 12:29:12.017011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.016975    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:12.017104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.017041    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:12.017104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.017071    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:12.322144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.322036    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" event={"ID":"038643c5-8fdf-4b50-bad8-4ee059945d89","Type":"ContainerStarted","Data":"e8ef39332a855587f478d307f1cfaf0d8a0e2b063b5918827cd1c4d568166878"}
Nov 03 12:29:12.322144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.322140    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" event={"ID":"038643c5-8fdf-4b50-bad8-4ee059945d89","Type":"ContainerStarted","Data":"1e02cc3361ed5f3df5987c88cd23468ea40180e75db38e1076946dba1bd4e59f"}
Nov 03 12:29:12.459459 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.459345    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podStartSLOduration=2.459330699 podStartE2EDuration="2.459330699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:12.458353584 +0000 UTC m=+2258.900555909" watchObservedRunningTime="2026-11-03 12:29:12.459330699 +0000 UTC m=+2258.901533003"
Nov 03 12:29:12.600591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.600489    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:12.938402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:12.935277    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:13.148958 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:13.148919    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:29:13.149460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:13.149137    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy" containerID="cri-o://aace5f634c61a618aa6c3f72fdd6ddac54f0d3a023640c7b02408b41b1a01322" gracePeriod=30
Nov 03 12:29:13.601116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:13.601087    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:13.601301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:13.601149    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:14.600033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:14.599995    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:14.600494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:14.600057    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:14.748286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:14.748242    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:14.897091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:14.897060    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:15.600013 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:15.599975    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:15.600191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:15.600049    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:16.599526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:16.599489    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:16.599847 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:16.599543    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:17.600126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:17.600087    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:17.600563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:17.600147    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:17.979271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:17.979230    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:18.191944 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.191912    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:18.390033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.389993    5784 generic.go:356] "Generic (PLEG): container finished" podID="d08c6341-545b-4f74-ad70-e93fccf83301" containerID="aace5f634c61a618aa6c3f72fdd6ddac54f0d3a023640c7b02408b41b1a01322" exitCode=0
Nov 03 12:29:18.390214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.390062    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" event={"ID":"d08c6341-545b-4f74-ad70-e93fccf83301","Type":"ContainerDied","Data":"aace5f634c61a618aa6c3f72fdd6ddac54f0d3a023640c7b02408b41b1a01322"}
Nov 03 12:29:18.442771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.442734    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:29:18.487072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487022    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487076    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487114    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487166    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487196    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487253    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487332    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487470    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.487678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487516    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\") pod \"d08c6341-545b-4f74-ad70-e93fccf83301\" (UID: \"d08c6341-545b-4f74-ad70-e93fccf83301\") "
Nov 03 12:29:18.488415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.487970    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:18.488415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.488224    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:18.488415 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.488241    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:18.488686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.488621    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:18.490202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.489938    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:18.490519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.490487    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:18.490638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.490539    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:29:18.491042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.491014    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:29:18.492005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.491984    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6" pod "d08c6341-545b-4f74-ad70-e93fccf83301" (UID: "d08c6341-545b-4f74-ad70-e93fccf83301"). InnerVolumeSpecName "kube-api-access-cwpk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589549    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpk6\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-kube-api-access-cwpk6\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589575    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d08c6341-545b-4f74-ad70-e93fccf83301-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589585    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d08c6341-545b-4f74-ad70-e93fccf83301-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589594    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589603    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589611    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589621    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589629    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d08c6341-545b-4f74-ad70-e93fccf83301-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.589990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.589637    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08c6341-545b-4f74-ad70-e93fccf83301-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:18.600181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.599671    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 12:29:18.600181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:18.599728    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 12:29:19.400436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.400404    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"
Nov 03 12:29:19.400644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.400433    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f" event={"ID":"d08c6341-545b-4f74-ad70-e93fccf83301","Type":"ContainerDied","Data":"1ff551c5c4d8e6efaf25a99153440c12abb5fd730917962d05fe3ecc9354991e"}
Nov 03 12:29:19.400644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.400517    5784 scope.go:122] "RemoveContainer" containerID="aace5f634c61a618aa6c3f72fdd6ddac54f0d3a023640c7b02408b41b1a01322"
Nov 03 12:29:19.602132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.602093    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:29:19.642102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.642070    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:19.642445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.642423    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:19.643319 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.643297    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:29:19.649707 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.649678    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-m926f"]
Nov 03 12:29:19.682433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:19.682348    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" path="/var/lib/kubelet/pods/d08c6341-545b-4f74-ad70-e93fccf83301/volumes"
Nov 03 12:29:20.652561 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:20.652521    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:21.265389 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:21.265348    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:21.431487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:21.431417    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:21.698577 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:21.698535    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:24.110749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:24.110718    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:26.481779 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:29:26.481660    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:29:27.334673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:27.334623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:28.456206 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:28.456174    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:28.610026 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:28.609993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:28.757894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:28.757814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:32.727949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:32.727887    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:33.407558 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:33.407524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:39.129240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:39.129200    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:39.431274 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:39.431193    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:39.729453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:39.729339    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:40.102730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:40.102645    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:40.703943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:40.703907    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:43.827214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:43.827177    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:43.993810 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:43.993777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:44.159805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:44.159745    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:45.744705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:45.744673    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:46.906486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:46.906439    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:47.067850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:47.067819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:48.759756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:48.759724    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:48.913076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:48.913038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:49.066713 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:49.066627    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:53.198854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:53.198820    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:53.339409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:53.339364    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:54.456525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:54.456461    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:55.372628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.372597    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:29:55.372884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.372841    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" podUID="f187e8ec-c241-4d73-ba45-84f7a4843da3" containerName="maas-api" containerID="cri-o://7735cdeaf0e88129c7e3c1db20d7ea172bc22746df32f6d727ac1b51f02ca54f" gracePeriod=30
Nov 03 12:29:55.527085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.527054    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:55.719511 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.718430    5784 generic.go:356] "Generic (PLEG): container finished" podID="f187e8ec-c241-4d73-ba45-84f7a4843da3" containerID="7735cdeaf0e88129c7e3c1db20d7ea172bc22746df32f6d727ac1b51f02ca54f" exitCode=0
Nov 03 12:29:55.722869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.722834    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" event={"ID":"f187e8ec-c241-4d73-ba45-84f7a4843da3","Type":"ContainerDied","Data":"7735cdeaf0e88129c7e3c1db20d7ea172bc22746df32f6d727ac1b51f02ca54f"}
Nov 03 12:29:55.750598 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.750526    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:29:55.860081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.855732    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") pod \"f187e8ec-c241-4d73-ba45-84f7a4843da3\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") "
Nov 03 12:29:55.860081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.855864    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\" (UniqueName: \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\") pod \"f187e8ec-c241-4d73-ba45-84f7a4843da3\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") "
Nov 03 12:29:55.860081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.855953    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\") pod \"f187e8ec-c241-4d73-ba45-84f7a4843da3\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") "
Nov 03 12:29:55.860081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.856070    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\") pod \"f187e8ec-c241-4d73-ba45-84f7a4843da3\" (UID: \"f187e8ec-c241-4d73-ba45-84f7a4843da3\") "
Nov 03 12:29:55.860081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.856919    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca" pod "f187e8ec-c241-4d73-ba45-84f7a4843da3" (UID: "f187e8ec-c241-4d73-ba45-84f7a4843da3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:29:55.868730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.868687    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh" pod "f187e8ec-c241-4d73-ba45-84f7a4843da3" (UID: "f187e8ec-c241-4d73-ba45-84f7a4843da3"). InnerVolumeSpecName "kube-api-access-rfnmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:29:55.869714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.869690    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs" pod "f187e8ec-c241-4d73-ba45-84f7a4843da3" (UID: "f187e8ec-c241-4d73-ba45-84f7a4843da3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:55.878131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.877909    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls" pod "f187e8ec-c241-4d73-ba45-84f7a4843da3" (UID: "f187e8ec-c241-4d73-ba45-84f7a4843da3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:29:55.959054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.958989    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f187e8ec-c241-4d73-ba45-84f7a4843da3-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:55.959370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.959343    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:55.959528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.959512    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfnmh\" (UniqueName: \"kubernetes.io/projected/f187e8ec-c241-4d73-ba45-84f7a4843da3-kube-api-access-rfnmh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:55.959616 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:55.959604    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f187e8ec-c241-4d73-ba45-84f7a4843da3-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:29:56.333077 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.333026    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:29:56.334394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.334174    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy"
Nov 03 12:29:56.334568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.334547    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy"
Nov 03 12:29:56.334696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.334683    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f187e8ec-c241-4d73-ba45-84f7a4843da3" containerName="maas-api"
Nov 03 12:29:56.334785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.334773    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f187e8ec-c241-4d73-ba45-84f7a4843da3" containerName="maas-api"
Nov 03 12:29:56.335051 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.335035    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08c6341-545b-4f74-ad70-e93fccf83301" containerName="istio-proxy"
Nov 03 12:29:56.335166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.335154    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f187e8ec-c241-4d73-ba45-84f7a4843da3" containerName="maas-api"
Nov 03 12:29:56.340659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.339919    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.465270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.465173    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c65m\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.465550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.465526    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.465785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.465766    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.465989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.465972    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.466170 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.466152    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.466550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.466410    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.466946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.466749    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.484187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.468348    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.484187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.481656    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.496398 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:29:56.496349    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:29:56.537400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.536625    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:29:56.582981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.582947    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583002    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583038    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583056    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583120    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583173    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583214    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c65m\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583247    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583279    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583432    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583483    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583796    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583837    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.583906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.583879    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.585717 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.585688    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.585903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.585887    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.636349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.636315    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.637045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.637016    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c65m\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.669424 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.669364    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:56.731655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.731457    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc" event={"ID":"f187e8ec-c241-4d73-ba45-84f7a4843da3","Type":"ContainerDied","Data":"2140f27cd0ab7d123de18aa471197b3f3f7be2ebfd7fb7fc3f27a9147e26b3e2"}
Nov 03 12:29:56.731655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.731483    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"
Nov 03 12:29:56.731655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.731524    5784 scope.go:122] "RemoveContainer" containerID="7735cdeaf0e88129c7e3c1db20d7ea172bc22746df32f6d727ac1b51f02ca54f"
Nov 03 12:29:56.890012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.889971    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:29:56.894532 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:56.894505    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-607dc3-5b8f6c8df9-s9qnc"]
Nov 03 12:29:57.100122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.100090    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:29:57.100613 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:29:57.100584    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2736721b_da3a_4fd0_a986_21e47e01765c.slice/crio-be491c4c01e62c8dbaa028d3609b4ed02a39225b882bb9579386286400cf7de2 WatchSource:0}: Error finding container be491c4c01e62c8dbaa028d3609b4ed02a39225b882bb9579386286400cf7de2: Status 404 returned error can't find the container with id be491c4c01e62c8dbaa028d3609b4ed02a39225b882bb9579386286400cf7de2
Nov 03 12:29:57.103361 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.103326    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:57.103466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.103425    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:57.103466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.103461    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:29:57.164775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.164746    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:29:57.682699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.682665    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f187e8ec-c241-4d73-ba45-84f7a4843da3" path="/var/lib/kubelet/pods/f187e8ec-c241-4d73-ba45-84f7a4843da3/volumes"
Nov 03 12:29:57.753826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.753792    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" event={"ID":"2736721b-da3a-4fd0-a986-21e47e01765c","Type":"ContainerStarted","Data":"04b51835a7e8a1bff684182fe46077dc51f95a772e08a4e6c2c4fa3ab3fb2b5a"}
Nov 03 12:29:57.754040 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.753854    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" event={"ID":"2736721b-da3a-4fd0-a986-21e47e01765c","Type":"ContainerStarted","Data":"be491c4c01e62c8dbaa028d3609b4ed02a39225b882bb9579386286400cf7de2"}
Nov 03 12:29:57.862912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:57.862857    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podStartSLOduration=1.862841838 podStartE2EDuration="1.862841838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:29:57.860688789 +0000 UTC m=+2304.302891119" watchObservedRunningTime="2026-11-03 12:29:57.862841838 +0000 UTC m=+2304.305044143"
Nov 03 12:29:58.670440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:58.670370    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:29:58.671728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:58.671699    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:29:58.671880 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:58.671748    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:29:59.670512 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:59.670479    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:29:59.670939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:29:59.670530    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:30:00.180143 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.180105    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:00.184684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.184658    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.244773 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.239744    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-w7dq8\""
Nov 03 12:30:00.349421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.349341    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtt7\" (UniqueName: \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.349821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.349797    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.373017 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.372964    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:00.450698 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.450597    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtt7\" (UniqueName: \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.450884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.450718    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.451356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.451335    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.504013 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.503982    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtt7\" (UniqueName: \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\") pod \"maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:00.670120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.670080    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:30:00.670302 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.670144    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:30:00.800638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:00.800557    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:01.202539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.202499    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:01.207445 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:30:01.207415    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4162caae_c21e_48a4_a88f_16898a7f4bb5.slice/crio-4e8e1b202ef0309a4d3bb546cde5b1b8ee7d599bb2fe51217c8b14a950ba1e38 WatchSource:0}: Error finding container 4e8e1b202ef0309a4d3bb546cde5b1b8ee7d599bb2fe51217c8b14a950ba1e38: Status 404 returned error can't find the container with id 4e8e1b202ef0309a4d3bb546cde5b1b8ee7d599bb2fe51217c8b14a950ba1e38
Nov 03 12:30:01.615627 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.615537    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:01.670597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.670570    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:30:01.670736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.670619    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:30:01.772116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.772086    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:01.783338 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.783305    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerStarted","Data":"69822522293f296091adc376b486daf89c456de2e9a97888270162d7ec285f40"}
Nov 03 12:30:01.783509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.783366    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerStarted","Data":"4e8e1b202ef0309a4d3bb546cde5b1b8ee7d599bb2fe51217c8b14a950ba1e38"}
Nov 03 12:30:01.889862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:01.889812    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" podStartSLOduration=1.889794523 podStartE2EDuration="1.889794523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:01.888562179 +0000 UTC m=+2308.330764484" watchObservedRunningTime="2026-11-03 12:30:01.889794523 +0000 UTC m=+2308.331996836"
Nov 03 12:30:02.669931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:02.669896    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:30:02.670117 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:02.669953    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:30:02.792055 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:02.792020    5784 generic.go:356] "Generic (PLEG): container finished" podID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerID="69822522293f296091adc376b486daf89c456de2e9a97888270162d7ec285f40" exitCode=60
Nov 03 12:30:02.792247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:02.792051    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerDied","Data":"69822522293f296091adc376b486daf89c456de2e9a97888270162d7ec285f40"}
Nov 03 12:30:02.792352 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:02.792336    5784 scope.go:122] "RemoveContainer" containerID="69822522293f296091adc376b486daf89c456de2e9a97888270162d7ec285f40"
Nov 03 12:30:03.239904 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.239870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:03.670144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.670113    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Nov 03 12:30:03.670308 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.670167    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Nov 03 12:30:03.742164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.742121    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:03.814138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.814103    5784 generic.go:356] "Generic (PLEG): container finished" podID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerID="0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c" exitCode=60
Nov 03 12:30:03.815158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.814744    5784 scope.go:122] "RemoveContainer" containerID="0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c"
Nov 03 12:30:03.815673 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:03.815650    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp_odh-ai-gateway-infra(4162caae-c21e-48a4-a88f-16898a7f4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5"
Nov 03 12:30:03.816008 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.814984    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerDied","Data":"0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c"}
Nov 03 12:30:03.816212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:03.816196    5784 scope.go:122] "RemoveContainer" containerID="69822522293f296091adc376b486daf89c456de2e9a97888270162d7ec285f40"
Nov 03 12:30:04.570962 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:04.570926    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:04.674311 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:04.674281    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:30:04.675036 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:04.675004    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:30:04.675692 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:04.675669    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:30:04.823728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:04.823649    5784 scope.go:122] "RemoveContainer" containerID="0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c"
Nov 03 12:30:04.823984 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:04.823960    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp_odh-ai-gateway-infra(4162caae-c21e-48a4-a88f-16898a7f4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5"
Nov 03 12:30:05.258993 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:05.258958    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:05.425195 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:05.425149    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:06.350826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:06.350787    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:06.659553 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:06.659523    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:07.142955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.142902    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:07.599659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.599553    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:30:07.600124 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.599791    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery" containerID="cri-o://761b33244b8c9daf0fc4fd3868528db2f2c0cfbafbb9d0d6cc5e4a8676540ced" gracePeriod=30
Nov 03 12:30:07.848237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.848209    5784 generic.go:356] "Generic (PLEG): container finished" podID="5e6d3196-245a-456c-b905-82142206ab5e" containerID="761b33244b8c9daf0fc4fd3868528db2f2c0cfbafbb9d0d6cc5e4a8676540ced" exitCode=0
Nov 03 12:30:07.848431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.848288    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" event={"ID":"5e6d3196-245a-456c-b905-82142206ab5e","Type":"ContainerDied","Data":"761b33244b8c9daf0fc4fd3868528db2f2c0cfbafbb9d0d6cc5e4a8676540ced"}
Nov 03 12:30:07.873651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.873611    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:30:07.971265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:07.971234    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:08.040850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.040818    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.040902    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.040957    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.041002    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.041040    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.041576    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:08.041764 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.041748    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.041910 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.041893    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\") pod \"5e6d3196-245a-456c-b905-82142206ab5e\" (UID: \"5e6d3196-245a-456c-b905-82142206ab5e\") "
Nov 03 12:30:08.042756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.042730    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-ca-configmap\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.044428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044356    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:08.044428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044413    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:08.044590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044479    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:08.044590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044553    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:08.044832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044808    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:08.044923 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.044905    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z" pod "5e6d3196-245a-456c-b905-82142206ab5e" (UID: "5e6d3196-245a-456c-b905-82142206ab5e"). InnerVolumeSpecName "kube-api-access-vxz7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:08.144346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144313    5784 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-cacerts\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.144346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144347    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.144565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144358    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxz7z\" (UniqueName: \"kubernetes.io/projected/5e6d3196-245a-456c-b905-82142206ab5e-kube-api-access-vxz7z\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.144565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144368    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-csr-dns-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.144565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144392    5784 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e6d3196-245a-456c-b905-82142206ab5e-local-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.144565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.144403    5784 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5e6d3196-245a-456c-b905-82142206ab5e-istio-kubeconfig\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:08.856852 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.856823    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"
Nov 03 12:30:08.856852 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.856840    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" event={"ID":"5e6d3196-245a-456c-b905-82142206ab5e","Type":"ContainerDied","Data":"bb03bb62bea65f15c2a54194b40528b64256b52abad7f6edc99740992e660c76"}
Nov 03 12:30:08.857422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:08.856925    5784 scope.go:122] "RemoveContainer" containerID="761b33244b8c9daf0fc4fd3868528db2f2c0cfbafbb9d0d6cc5e4a8676540ced"
Nov 03 12:30:09.055567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:09.055529    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:30:09.101631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:09.101599    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s"]
Nov 03 12:30:09.681393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:09.681352    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6d3196-245a-456c-b905-82142206ab5e" path="/var/lib/kubelet/pods/5e6d3196-245a-456c-b905-82142206ab5e/volumes"
Nov 03 12:30:10.432514 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:10.432484    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:11.233035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:11.233004    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:11.771873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:11.771839    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:11.921841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:11.921798    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:12.848101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:12.848063    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bhj8s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:30:12.848534 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:12.848125    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bhj8s" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.85:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:30:13.970387 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:13.970342    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:14.118238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:14.118205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:15.674830 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:15.674794    5784 scope.go:122] "RemoveContainer" containerID="0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c"
Nov 03 12:30:16.197606 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.197529    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:16.603624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.603551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:16.916552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.916520    5784 generic.go:356] "Generic (PLEG): container finished" podID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerID="11641d73e527fb0ded4f112645b8f4bd6fc78572ff891e87ea489385ff21bdbf" exitCode=60
Nov 03 12:30:16.917041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.916590    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerDied","Data":"11641d73e527fb0ded4f112645b8f4bd6fc78572ff891e87ea489385ff21bdbf"}
Nov 03 12:30:16.917041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.916651    5784 scope.go:122] "RemoveContainer" containerID="0acf1db31f5e168e3d18ae91f040b9898c19b1631a5b8ee744ac7d6b760db21c"
Nov 03 12:30:16.917176 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:16.917094    5784 scope.go:122] "RemoveContainer" containerID="11641d73e527fb0ded4f112645b8f4bd6fc78572ff891e87ea489385ff21bdbf"
Nov 03 12:30:16.917496 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:16.917463    5784 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp_odh-ai-gateway-infra(4162caae-c21e-48a4-a88f-16898a7f4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5"
Nov 03 12:30:18.087444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.087408    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:18.311799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.311777    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:18.390995 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.390965    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\" (UniqueName: \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\") pod \"4162caae-c21e-48a4-a88f-16898a7f4bb5\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") "
Nov 03 12:30:18.391169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.391109    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\") pod \"4162caae-c21e-48a4-a88f-16898a7f4bb5\" (UID: \"4162caae-c21e-48a4-a88f-16898a7f4bb5\") "
Nov 03 12:30:18.391488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.391462    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca" pod "4162caae-c21e-48a4-a88f-16898a7f4bb5" (UID: "4162caae-c21e-48a4-a88f-16898a7f4bb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:18.393236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.393204    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7" pod "4162caae-c21e-48a4-a88f-16898a7f4bb5" (UID: "4162caae-c21e-48a4-a88f-16898a7f4bb5"). InnerVolumeSpecName "kube-api-access-8mtt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:18.499354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.499313    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4162caae-c21e-48a4-a88f-16898a7f4bb5-maas-api-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.499354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.499353    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtt7\" (UniqueName: \"kubernetes.io/projected/4162caae-c21e-48a4-a88f-16898a7f4bb5-kube-api-access-8mtt7\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:18.941239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.941212    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"
Nov 03 12:30:18.941239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.941204    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp" event={"ID":"4162caae-c21e-48a4-a88f-16898a7f4bb5","Type":"ContainerDied","Data":"4e8e1b202ef0309a4d3bb546cde5b1b8ee7d599bb2fe51217c8b14a950ba1e38"}
Nov 03 12:30:18.941536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:18.941285    5784 scope.go:122] "RemoveContainer" containerID="11641d73e527fb0ded4f112645b8f4bd6fc78572ff891e87ea489385ff21bdbf"
Nov 03 12:30:19.095756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:19.095715    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:19.153213 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:19.153173    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5951bc-29895150-gm8pp"]
Nov 03 12:30:19.684681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:19.683518    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" path="/var/lib/kubelet/pods/4162caae-c21e-48a4-a88f-16898a7f4bb5/volumes"
Nov 03 12:30:19.951467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:19.951345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:22.320717 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:22.320682    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:22.480718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:22.480675    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:23.331655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:23.331611    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:26.474250 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:26.474209    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:30:27.169636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:27.169606    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:28.837669 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:28.837632    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:29.967673 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:29.967636    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:30:29.968130 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:29.967832    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" containerID="cri-o://5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d" gracePeriod=30
Nov 03 12:30:31.229989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:31.229948    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-m5cbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:30:31.230369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:31.230028    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:30:31.819168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:31.819114    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:34.792590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:34.792555    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:35.235943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.235913    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:30:35.394068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394038    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394096    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394145    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394173    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394223    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394299    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394339    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394368    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394505 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394454    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\") pod \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\" (UID: \"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9\") "
Nov 03 12:30:35.394877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.394850    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.395223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.395191    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:35.395363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.395344    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.395447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.395427    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.395562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.395539    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.397626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.397598    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:30:35.397722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.397675    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.397771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.397744    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "kube-api-access-xzpvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:35.398958 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.398936    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy" pod "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" (UID: "35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496265    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496305    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496319    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496342    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496357    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496370    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzpvk\" (UniqueName: \"kubernetes.io/projected/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-kube-api-access-xzpvk\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496414    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496431    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:35.496486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:35.496445    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:36.067066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.066973    5784 generic.go:356] "Generic (PLEG): container finished" podID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerID="5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d" exitCode=0
Nov 03 12:30:36.067066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.067044    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"
Nov 03 12:30:36.067609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.067061    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" event={"ID":"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9","Type":"ContainerDied","Data":"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"}
Nov 03 12:30:36.067609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.067128    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr" event={"ID":"35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9","Type":"ContainerDied","Data":"a0606c5566c17925e1a79f1e32ed4963ce5ed480dc85b2adaa96cabc6f8124a6"}
Nov 03 12:30:36.067609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.067159    5784 scope.go:122] "RemoveContainer" containerID="5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"
Nov 03 12:30:36.077191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.077172    5784 scope.go:122] "RemoveContainer" containerID="5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"
Nov 03 12:30:36.077464 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:36.077443    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d\": container with ID starting with 5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d not found: ID does not exist" containerID="5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"
Nov 03 12:30:36.077521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.077476    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d"} err="failed to get container status \"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d\": rpc error: code = NotFound desc = could not find container \"5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d\": container with ID starting with 5c7c86ee71e921fc548f27d83cd7c01e2d544a8cdf7599a0f3c8408423a4508d not found: ID does not exist"
Nov 03 12:30:36.265438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.265370    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:30:36.311436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:36.311404    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-m5cbr"]
Nov 03 12:30:37.682152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:37.682121    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" path="/var/lib/kubelet/pods/35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9/volumes"
Nov 03 12:30:37.732541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:37.732482    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:41.667571 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.667487    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:30:41.668228 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668207    5784 cpu_manager.go:414] "containerMap: removing container" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668232    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668287    5784 cpu_manager.go:414] "containerMap: removing container" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy"
Nov 03 12:30:41.668298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668294    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668302    5784 cpu_manager.go:414] "containerMap: removing container" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668307    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668325    5784 cpu_manager.go:414] "containerMap: removing container" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668329    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668338    5784 cpu_manager.go:414] "containerMap: removing container" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery"
Nov 03 12:30:41.668420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668342    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery"
Nov 03 12:30:41.668601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668448    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="35c89c38-35be-4b0b-8f63-e1a5b9ae7ee9" containerName="istio-proxy"
Nov 03 12:30:41.668601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668459    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668468    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.668601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668473    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6d3196-245a-456c-b905-82142206ab5e" containerName="discovery"
Nov 03 12:30:41.668601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.668480    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4162caae-c21e-48a4-a88f-16898a7f4bb5" containerName="cleanup"
Nov 03 12:30:41.672990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.672969    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.782181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.782146    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.782463 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.782438    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.783202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.782639    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.783202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783155    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.787551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783474    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.787551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783548    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.787551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783649    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.787551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783679    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.787551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.783721    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh4x8\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.866726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.866673    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.913309    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.914505    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.914628    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915459    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915533    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915573    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh4x8\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915638    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915684    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915713    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.915800    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.918460    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.919728    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.919894    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.920301    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.920639    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.921066    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.921825    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.925465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.923757    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.935154 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.934813    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:30:41.972274 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.972236    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh4x8\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.972444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.972238    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\") pod \"e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") " pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:41.987102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:41.987075    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:42.016813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.016786    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.016994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.016863    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dwtl\" (UniqueName: \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.016994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.016951    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.017112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.017018    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.034871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.034830    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2daac7-serving-cert\""
Nov 03 12:30:42.118536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.118495    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.118731 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:42.118636    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-serving-cert: secret "maas-api-e2e-ipp-b-2daac7-serving-cert" not found
Nov 03 12:30:42.118731 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:42.118695    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls podName:74336dec-d18f-4043-a09e-a010607e47ab nodeName:}" failed. No retries permitted until 2026-11-03 12:30:42.618680354 +0000 UTC m=+2349.060882637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls") pod "maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" (UID: "74336dec-d18f-4043-a09e-a010607e47ab") : secret "maas-api-e2e-ipp-b-2daac7-serving-cert" not found
Nov 03 12:30:42.118731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.118641    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dwtl\" (UniqueName: \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.118929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.118785    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.118929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.118851    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.119468 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.119448    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.121543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.121519    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.141552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.141521    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:42.173898 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.173867    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dwtl\" (UniqueName: \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.304119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.304038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:42.417023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.416991    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:30:42.418601 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:30:42.418575    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc696c1c7_0b7f_49f0_8aa5_c23f5467dd4d.slice/crio-21a6e7143d4127a49cb61fad9598a37b81380c67ace99b63705cbc60fedf7c56 WatchSource:0}: Error finding container 21a6e7143d4127a49cb61fad9598a37b81380c67ace99b63705cbc60fedf7c56: Status 404 returned error can't find the container with id 21a6e7143d4127a49cb61fad9598a37b81380c67ace99b63705cbc60fedf7c56
Nov 03 12:30:42.421166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.421131    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:30:42.421264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.421198    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:30:42.421264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.421233    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:30:42.462672 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.462583    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:42.625323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.623584    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.627005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.626974    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:42.853715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:42.853672    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:43.189459 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.187134    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" event={"ID":"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d","Type":"ContainerStarted","Data":"adcf9c00d19f0e0f2afa9e61c72271fc2265f9915eb51cd6bc4368aabefbd3e4"}
Nov 03 12:30:43.189459 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.187206    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" event={"ID":"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d","Type":"ContainerStarted","Data":"21a6e7143d4127a49cb61fad9598a37b81380c67ace99b63705cbc60fedf7c56"}
Nov 03 12:30:43.287020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.286960    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:30:43.352412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.338918    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podStartSLOduration=2.33889678 podStartE2EDuration="2.33889678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:43.29666603 +0000 UTC m=+2349.738868346" watchObservedRunningTime="2026-11-03 12:30:43.33889678 +0000 UTC m=+2349.781099085"
Nov 03 12:30:43.988237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.988189    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:43.989775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.989745    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:43.989924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:43.989794    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:44.197315 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:44.197275    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" event={"ID":"74336dec-d18f-4043-a09e-a010607e47ab","Type":"ContainerStarted","Data":"03d98d68e90e7580786b5e09ea496c524acd7f8b9cb5657b4c731025424531f6"}
Nov 03 12:30:44.988081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:44.988051    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:44.988265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:44.988104    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:45.204880 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:45.204847    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" event={"ID":"74336dec-d18f-4043-a09e-a010607e47ab","Type":"ContainerStarted","Data":"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"}
Nov 03 12:30:45.205084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:45.204909    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:45.312503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:45.312368    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" podStartSLOduration=3.307507858 podStartE2EDuration="4.312354995s" totalImagesPullingTime="1.004847137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:30:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:30:45.309837144 +0000 UTC m=+2351.752039451" watchObservedRunningTime="2026-11-03 12:30:45.312354995 +0000 UTC m=+2351.754557299"
Nov 03 12:30:45.987752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:45.987722    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:45.987942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:45.987780    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:46.838402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:46.835817    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:46.987566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:46.987530    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:46.987766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:46.987595    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:47.240287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:47.240161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:47.987766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:47.987730    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:47.988155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:47.987787    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:48.988280 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:48.988242    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 12:30:48.988723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:48.988299    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 12:30:49.992864 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:49.992832    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:49.993349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:49.993236    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:49.994019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:49.993999    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:30:50.731152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:50.731115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:51.200277 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.200239    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:51.216921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.216897    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:30:51.358260 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.358221    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:51.751309 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.751268    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:30:51.751628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.751532    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" podUID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" containerName="maas-api" containerID="cri-o://04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d" gracePeriod=30
Nov 03 12:30:51.873564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:51.873531    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:52.051256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.051230    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:30:52.204689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.204648    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\") pod \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") "
Nov 03 12:30:52.205184 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.204784    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") pod \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") "
Nov 03 12:30:52.205184 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.204953    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\" (UniqueName: \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\") pod \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") "
Nov 03 12:30:52.205184 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.205010    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\") pod \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\" (UID: \"4aa19ce4-982a-41f0-9f82-5d2c2e973844\") "
Nov 03 12:30:52.205551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.205501    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca" pod "4aa19ce4-982a-41f0-9f82-5d2c2e973844" (UID: "4aa19ce4-982a-41f0-9f82-5d2c2e973844"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:30:52.207225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.207199    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs" pod "4aa19ce4-982a-41f0-9f82-5d2c2e973844" (UID: "4aa19ce4-982a-41f0-9f82-5d2c2e973844"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:52.207457 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.207426    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z" pod "4aa19ce4-982a-41f0-9f82-5d2c2e973844" (UID: "4aa19ce4-982a-41f0-9f82-5d2c2e973844"). InnerVolumeSpecName "kube-api-access-fzb9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:30:52.207570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.207520    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls" pod "4aa19ce4-982a-41f0-9f82-5d2c2e973844" (UID: "4aa19ce4-982a-41f0-9f82-5d2c2e973844"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:30:52.249012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.248976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:52.282144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.282048    5784 generic.go:356] "Generic (PLEG): container finished" podID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" containerID="04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d" exitCode=0
Nov 03 12:30:52.282455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.282189    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" event={"ID":"4aa19ce4-982a-41f0-9f82-5d2c2e973844","Type":"ContainerDied","Data":"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"}
Nov 03 12:30:52.282455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.282241    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2" event={"ID":"4aa19ce4-982a-41f0-9f82-5d2c2e973844","Type":"ContainerDied","Data":"ef83a5e70a6e19b213e4614fd3f7f5b756c83cb062c955ece6e57672ac8b3650"}
Nov 03 12:30:52.282455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.282278    5784 scope.go:122] "RemoveContainer" containerID="04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"
Nov 03 12:30:52.282827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.282481    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"
Nov 03 12:30:52.297051 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.297033    5784 scope.go:122] "RemoveContainer" containerID="04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"
Nov 03 12:30:52.297367 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:52.297349    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d\": container with ID starting with 04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d not found: ID does not exist" containerID="04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"
Nov 03 12:30:52.297459 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.297410    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d"} err="failed to get container status \"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d\": rpc error: code = NotFound desc = could not find container \"04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d\": container with ID starting with 04a8af1f78186d2ca9b5601528e06e6aad0b9c57a2499eb0b20f6d265625b60d not found: ID does not exist"
Nov 03 12:30:52.307155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.307131    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzb9z\" (UniqueName: \"kubernetes.io/projected/4aa19ce4-982a-41f0-9f82-5d2c2e973844-kube-api-access-fzb9z\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:52.307155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.307156    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4aa19ce4-982a-41f0-9f82-5d2c2e973844-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:52.307283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.307167    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:52.307283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.307176    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4aa19ce4-982a-41f0-9f82-5d2c2e973844-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:30:52.436134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.436100    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:30:52.440326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:52.440290    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-40ffcd-55885d9cb-m7gf2"]
Nov 03 12:30:53.534945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:53.534910    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:53.678024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:53.677987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:53.682509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:53.682482    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" path="/var/lib/kubelet/pods/4aa19ce4-982a-41f0-9f82-5d2c2e973844/volumes"
Nov 03 12:30:56.476658 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:30:56.476622    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:30:56.769199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:56.769115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:56.928989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:56.928944    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:57.352341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:57.352306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:57.530573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:57.530535    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:57.716363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:57.713619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:58.235179 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:58.235140    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:58.859172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:58.859134    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:59.131978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:59.131942    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:30:59.988187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:30:59.988151    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:00.520911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:00.520875    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:01.206503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:01.206469    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:01.645458 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:01.645426    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:01.815212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:01.815180    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:03.064386 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:03.064349    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:03.514290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:03.514261    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:04.067580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:04.067545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:07.399108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:07.399077    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:10.599999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:10.599963    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:15.486814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:15.486782    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:19.330414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.330354    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:31:19.330935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.330574    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" podUID="ed620fcf-e6e7-4848-9631-8cfe6c639307" containerName="maas-api" containerID="cri-o://d4310fa365511e2599bdff00fe1bbb485b863404f1e155ec5d1fda46a124f1b3" gracePeriod=30
Nov 03 12:31:19.463091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.463057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:19.476132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.476089    5784 generic.go:356] "Generic (PLEG): container finished" podID="ed620fcf-e6e7-4848-9631-8cfe6c639307" containerID="d4310fa365511e2599bdff00fe1bbb485b863404f1e155ec5d1fda46a124f1b3" exitCode=0
Nov 03 12:31:19.476322 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.476261    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" event={"ID":"ed620fcf-e6e7-4848-9631-8cfe6c639307","Type":"ContainerDied","Data":"d4310fa365511e2599bdff00fe1bbb485b863404f1e155ec5d1fda46a124f1b3"}
Nov 03 12:31:19.643116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.643082    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:31:19.803583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.803550    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") pod \"ed620fcf-e6e7-4848-9631-8cfe6c639307\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") "
Nov 03 12:31:19.803809 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.803647    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\") pod \"ed620fcf-e6e7-4848-9631-8cfe6c639307\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") "
Nov 03 12:31:19.803809 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.803701    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\") pod \"ed620fcf-e6e7-4848-9631-8cfe6c639307\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") "
Nov 03 12:31:19.803939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.803822    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\" (UniqueName: \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\") pod \"ed620fcf-e6e7-4848-9631-8cfe6c639307\" (UID: \"ed620fcf-e6e7-4848-9631-8cfe6c639307\") "
Nov 03 12:31:19.804942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.804881    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca" pod "ed620fcf-e6e7-4848-9631-8cfe6c639307" (UID: "ed620fcf-e6e7-4848-9631-8cfe6c639307"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:19.810797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.810747    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv" pod "ed620fcf-e6e7-4848-9631-8cfe6c639307" (UID: "ed620fcf-e6e7-4848-9631-8cfe6c639307"). InnerVolumeSpecName "kube-api-access-dz8nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:19.814451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.812670    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs" pod "ed620fcf-e6e7-4848-9631-8cfe6c639307" (UID: "ed620fcf-e6e7-4848-9631-8cfe6c639307"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:19.824010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.823977    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls" pod "ed620fcf-e6e7-4848-9631-8cfe6c639307" (UID: "ed620fcf-e6e7-4848-9631-8cfe6c639307"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:19.824618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.824589    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:19.824741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.824626    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed620fcf-e6e7-4848-9631-8cfe6c639307-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:19.824741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.824637    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8nv\" (UniqueName: \"kubernetes.io/projected/ed620fcf-e6e7-4848-9631-8cfe6c639307-kube-api-access-dz8nv\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:19.824741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.824646    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed620fcf-e6e7-4848-9631-8cfe6c639307-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:19.952715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:19.952683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:20.110996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.110962    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:20.486680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.486655    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"
Nov 03 12:31:20.487106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.486677    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz" event={"ID":"ed620fcf-e6e7-4848-9631-8cfe6c639307","Type":"ContainerDied","Data":"1ffda19072a3a3e5c86dc46b7d1fa4f0cf694a47513423eb232d9b83a4ffefce"}
Nov 03 12:31:20.487106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.486760    5784 scope.go:122] "RemoveContainer" containerID="d4310fa365511e2599bdff00fe1bbb485b863404f1e155ec5d1fda46a124f1b3"
Nov 03 12:31:20.641661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.641630    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:31:20.687771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:20.687741    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d25504-6d89d67f77-q6wpz"]
Nov 03 12:31:21.138276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:21.138243    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:21.682011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:21.681972    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed620fcf-e6e7-4848-9631-8cfe6c639307" path="/var/lib/kubelet/pods/ed620fcf-e6e7-4848-9631-8cfe6c639307/volumes"
Nov 03 12:31:22.020064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:22.019952    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:22.193584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:22.193551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:23.435716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:23.435678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:23.949908 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:23.949868    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:25.615923 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:25.615876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:25.937018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:25.936978    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:26.107894 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:26.107859    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:26.289425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:26.289279    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:26.472589 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:31:26.472554    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:31:29.864217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:29.864168    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:30.071721 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:30.071678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:30.236032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:30.235996    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:30.914396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:30.911887    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:34.126252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:34.126222    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:31:34.140018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:34.139991    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:31:35.522960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:35.522920    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:35.764339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:35.764295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:35.930432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:35.930347    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:36.530279 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:36.530233    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:37.748116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:37.748080    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:39.252515 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:39.252474    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:39.399230 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:39.399199    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:40.007194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:40.007141    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:43.466749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:43.466713    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:43.746254 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:43.746140    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:31:43.746864 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:43.746415    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" podUID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" containerName="maas-api" containerID="cri-o://6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9" gracePeriod=30
Nov 03 12:31:43.874484 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:43.874344    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:44.080146 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.079845    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:31:44.167803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.164980    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") pod \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") "
Nov 03 12:31:44.167803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.165079    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\") pod \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") "
Nov 03 12:31:44.167803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.165113    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\" (UniqueName: \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\") pod \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") "
Nov 03 12:31:44.167803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.165233    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\") pod \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\" (UID: \"622b8886-e59b-4bb2-9e6c-d83372b89eb0\") "
Nov 03 12:31:44.167803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.165811    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca" pod "622b8886-e59b-4bb2-9e6c-d83372b89eb0" (UID: "622b8886-e59b-4bb2-9e6c-d83372b89eb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:31:44.168862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.168829    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls" pod "622b8886-e59b-4bb2-9e6c-d83372b89eb0" (UID: "622b8886-e59b-4bb2-9e6c-d83372b89eb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:44.174817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.170478    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs" pod "622b8886-e59b-4bb2-9e6c-d83372b89eb0" (UID: "622b8886-e59b-4bb2-9e6c-d83372b89eb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:31:44.175119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.175070    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6" pod "622b8886-e59b-4bb2-9e6c-d83372b89eb0" (UID: "622b8886-e59b-4bb2-9e6c-d83372b89eb0"). InnerVolumeSpecName "kube-api-access-hdtr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:31:44.267347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.267289    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.267347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.267328    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/622b8886-e59b-4bb2-9e6c-d83372b89eb0-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.267347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.267346    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdtr6\" (UniqueName: \"kubernetes.io/projected/622b8886-e59b-4bb2-9e6c-d83372b89eb0-kube-api-access-hdtr6\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.267675 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.267364    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/622b8886-e59b-4bb2-9e6c-d83372b89eb0-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:31:44.395325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.395273    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:44.540096 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.540053    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:44.679846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.679758    5784 generic.go:356] "Generic (PLEG): container finished" podID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" containerID="6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9" exitCode=0
Nov 03 12:31:44.679846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.679802    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"
Nov 03 12:31:44.680048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.679844    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" event={"ID":"622b8886-e59b-4bb2-9e6c-d83372b89eb0","Type":"ContainerDied","Data":"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"}
Nov 03 12:31:44.680048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.679914    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs" event={"ID":"622b8886-e59b-4bb2-9e6c-d83372b89eb0","Type":"ContainerDied","Data":"55586e38d266231ba5779c4ec4994145011e2f050dbbcc0526505d3b530581f2"}
Nov 03 12:31:44.680048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.679946    5784 scope.go:122] "RemoveContainer" containerID="6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"
Nov 03 12:31:44.692489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.692462    5784 scope.go:122] "RemoveContainer" containerID="6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"
Nov 03 12:31:44.692998 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:31:44.692843    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9\": container with ID starting with 6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9 not found: ID does not exist" containerID="6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"
Nov 03 12:31:44.692998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.692873    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9"} err="failed to get container status \"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9\": rpc error: code = NotFound desc = could not find container \"6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9\": container with ID starting with 6a99445b10c31e934378dc5a1413501dc5665e2d1c6081d6d9b5e1a265fac7a9 not found: ID does not exist"
Nov 03 12:31:44.838972 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.838931    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:31:44.883919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:44.883890    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c02ae4-6d56c77d65-8xhxs"]
Nov 03 12:31:45.681696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:45.681667    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" path="/var/lib/kubelet/pods/622b8886-e59b-4bb2-9e6c-d83372b89eb0/volumes"
Nov 03 12:31:45.800702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:45.800669    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:49.002987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:49.002953    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:49.158169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:49.158135    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:50.428269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:50.428233    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:50.626226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:50.626171    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:50.968593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:50.968539    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:53.846759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:53.846721    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:55.021298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:55.021267    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:56.477870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:56.477800    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:56.543174 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:31:56.543138    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:31:57.751058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:57.751024    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:31:58.169702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.169661    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170340    5784 cpu_manager.go:414] "containerMap: removing container" podUID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170362    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170372    5784 cpu_manager.go:414] "containerMap: removing container" podUID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170393    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170422    5784 cpu_manager.go:414] "containerMap: removing container" podUID="ed620fcf-e6e7-4848-9631-8cfe6c639307" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170428    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed620fcf-e6e7-4848-9631-8cfe6c639307" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170510    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aa19ce4-982a-41f0-9f82-5d2c2e973844" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170519    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="622b8886-e59b-4bb2-9e6c-d83372b89eb0" containerName="maas-api"
Nov 03 12:31:58.170857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.170532    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed620fcf-e6e7-4848-9631-8cfe6c639307" containerName="maas-api"
Nov 03 12:31:58.173978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.173956    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279091    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279176    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279237    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw6r\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279281    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279441    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279566    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279676    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279760 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279727    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.279816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.279755    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.286071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.286046    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:31:58.324359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.324328    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9893f0-openshift-default-dockercfg-ptlmk\""
Nov 03 12:31:58.380887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.380847    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.380887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.380888    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.380939    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.380970    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381005    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw6r\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381033    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381166    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381217    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381310    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381331    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381362    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381620    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381864 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381841    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.381949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.381845    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.383681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.383658    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.383906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.383885    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.433674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.433586    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.434003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.433963    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw6r\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.488483 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.488448    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:31:58.906271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.906232    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:31:58.906854 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:31:58.906829    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fdf350_b561_4322_a869_d8ed0613ba76.slice/crio-41ca5b49da528eee9cc99d70375e3b95923f77138f0290831aeb77d34fd9ea7a WatchSource:0}: Error finding container 41ca5b49da528eee9cc99d70375e3b95923f77138f0290831aeb77d34fd9ea7a: Status 404 returned error can't find the container with id 41ca5b49da528eee9cc99d70375e3b95923f77138f0290831aeb77d34fd9ea7a
Nov 03 12:31:58.909775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.909731    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:31:58.909877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.909821    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:31:58.909877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:58.909868    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:31:59.808542 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:59.808506    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" event={"ID":"18fdf350-b561-4322-a869-d8ed0613ba76","Type":"ContainerStarted","Data":"9f1e8678d1a955d143b5b07f4934b37bc7457a84d14355d2923a3a26a48ab949"}
Nov 03 12:31:59.808735 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:59.808564    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" event={"ID":"18fdf350-b561-4322-a869-d8ed0613ba76","Type":"ContainerStarted","Data":"41ca5b49da528eee9cc99d70375e3b95923f77138f0290831aeb77d34fd9ea7a"}
Nov 03 12:31:59.965442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:59.965356    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podStartSLOduration=1.965342246 podStartE2EDuration="1.965342246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:31:59.961027878 +0000 UTC m=+2426.403230195" watchObservedRunningTime="2026-11-03 12:31:59.965342246 +0000 UTC m=+2426.407544550"
Nov 03 12:31:59.989928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:31:59.989895    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:00.488966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:00.488935    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:32:00.490359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:00.490333    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:00.490482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:00.490403    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:01.489110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:01.489067    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:01.489646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:01.489127    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:02.277959 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:02.277905    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:02.489097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:02.489063    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:02.489262 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:02.489110    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:03.489699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:03.489669    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:03.490132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:03.490108    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:03.547672 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:03.547642    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:03.991164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:03.991129    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:04.489537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:04.489507    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:04.489712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:04.489561    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:05.489631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:05.489594    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:05.490104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:05.489646    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:06.273709 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.273659    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:32:06.279087 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.279059    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.301313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.301280    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.301523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.301329    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnxt\" (UniqueName: \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.301523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.301416    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.301656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.301541    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.328780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.328744    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:32:06.370479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.370450    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-eb4a0f-serving-cert\""
Nov 03 12:32:06.403336 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.403304    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.403537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.403349    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnxt\" (UniqueName: \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.403537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.403413    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.403537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.403431    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.403537 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:06.403514    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-serving-cert: secret "maas-api-e2e-praxis-a-eb4a0f-serving-cert" not found
Nov 03 12:32:06.403755 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:06.403608    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls podName:8536dde7-8075-4743-bac5-050277e0b7cb nodeName:}" failed. No retries permitted until 2026-11-03 12:32:06.903586051 +0000 UTC m=+2433.345788338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls") pod "maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" (UID: "8536dde7-8075-4743-bac5-050277e0b7cb") : secret "maas-api-e2e-praxis-a-eb4a0f-serving-cert" not found
Nov 03 12:32:06.404165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.404140    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.405806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.405785    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.456611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.456580    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnxt\" (UniqueName: \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.488877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.488841    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 03 12:32:06.489050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.488905    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 03 12:32:06.489657 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.489630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:06.647516 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.647476    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:06.909869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.909767    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.913573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.913526    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:06.932173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:06.932138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:07.196235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.196151    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:07.493186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.493106    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:32:07.493681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.493510    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:32:07.494105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.494088    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:32:07.566414 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:32:07.566367    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8536dde7_8075_4743_bac5_050277e0b7cb.slice/crio-a18fd96a557df00eb40c0cff38d118bbaf80c1996d050cb0a66d180c5b1dc2bc WatchSource:0}: Error finding container a18fd96a557df00eb40c0cff38d118bbaf80c1996d050cb0a66d180c5b1dc2bc: Status 404 returned error can't find the container with id a18fd96a557df00eb40c0cff38d118bbaf80c1996d050cb0a66d180c5b1dc2bc
Nov 03 12:32:07.568268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.568249    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:32:07.590105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.590082    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:32:07.889349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:07.889307    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" event={"ID":"8536dde7-8075-4743-bac5-050277e0b7cb","Type":"ContainerStarted","Data":"a18fd96a557df00eb40c0cff38d118bbaf80c1996d050cb0a66d180c5b1dc2bc"}
Nov 03 12:32:08.897243 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:08.897211    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" event={"ID":"8536dde7-8075-4743-bac5-050277e0b7cb","Type":"ContainerStarted","Data":"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"}
Nov 03 12:32:08.897668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:08.897290    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:09.005054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:09.005004    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" podStartSLOduration=2.102253697 podStartE2EDuration="3.004991387s" totalImagesPullingTime="902.73769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:32:09.002861416 +0000 UTC m=+2435.445063721" watchObservedRunningTime="2026-11-03 12:32:09.004991387 +0000 UTC m=+2435.447193692"
Nov 03 12:32:11.068403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:11.068297    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:14.909391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:14.909343    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:32:15.615679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:15.615643    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:19.481638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:19.481587    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:19.946759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:19.946717    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:20.128862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:20.128824    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:20.301294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:20.301200    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:24.243433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:24.243371    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:24.403769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:24.403717    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:24.578791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:24.578701    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:26.473607 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:26.473544    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:32:27.083543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:27.083497    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:27.394670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:27.394630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:27.562400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:27.557124    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:28.196932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:28.196887    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:28.373098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:28.373050    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:29.808219 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:29.808185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:32.623650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:32.623612    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:32.788522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:32.788481    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:32.963828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:32.963800    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:34.637750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:34.637717    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:35.731043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:35.731007    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:35.887167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:35.887133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:38.492781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.492744    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:32:38.493356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.492984    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" podUID="74336dec-d18f-4043-a09e-a010607e47ab" containerName="maas-api" containerID="cri-o://e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95" gracePeriod=30
Nov 03 12:32:38.625118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.625088    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:38.789335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.789306    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:32:38.886808 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.886771    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\") pod \"74336dec-d18f-4043-a09e-a010607e47ab\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") "
Nov 03 12:32:38.886992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.886841    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\") pod \"74336dec-d18f-4043-a09e-a010607e47ab\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") "
Nov 03 12:32:38.886992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.886881    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") pod \"74336dec-d18f-4043-a09e-a010607e47ab\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") "
Nov 03 12:32:38.887116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.887007    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\" (UniqueName: \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\") pod \"74336dec-d18f-4043-a09e-a010607e47ab\" (UID: \"74336dec-d18f-4043-a09e-a010607e47ab\") "
Nov 03 12:32:38.887272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.887246    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca" pod "74336dec-d18f-4043-a09e-a010607e47ab" (UID: "74336dec-d18f-4043-a09e-a010607e47ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:32:38.887543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.887525    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74336dec-d18f-4043-a09e-a010607e47ab-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:38.889058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.889035    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls" pod "74336dec-d18f-4043-a09e-a010607e47ab" (UID: "74336dec-d18f-4043-a09e-a010607e47ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:38.889450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.889423    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs" pod "74336dec-d18f-4043-a09e-a010607e47ab" (UID: "74336dec-d18f-4043-a09e-a010607e47ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:32:38.889607 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.889579    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl" pod "74336dec-d18f-4043-a09e-a010607e47ab" (UID: "74336dec-d18f-4043-a09e-a010607e47ab"). InnerVolumeSpecName "kube-api-access-9dwtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:32:38.988187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.988150    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:38.988187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.988183    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74336dec-d18f-4043-a09e-a010607e47ab-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:38.988187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:38.988193    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dwtl\" (UniqueName: \"kubernetes.io/projected/74336dec-d18f-4043-a09e-a010607e47ab-kube-api-access-9dwtl\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:32:39.083902 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.083816    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:39.149563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.149534    5784 generic.go:356] "Generic (PLEG): container finished" podID="74336dec-d18f-4043-a09e-a010607e47ab" containerID="e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95" exitCode=0
Nov 03 12:32:39.149738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.149581    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"
Nov 03 12:32:39.149738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.149614    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" event={"ID":"74336dec-d18f-4043-a09e-a010607e47ab","Type":"ContainerDied","Data":"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"}
Nov 03 12:32:39.149738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.149672    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk" event={"ID":"74336dec-d18f-4043-a09e-a010607e47ab","Type":"ContainerDied","Data":"03d98d68e90e7580786b5e09ea496c524acd7f8b9cb5657b4c731025424531f6"}
Nov 03 12:32:39.149738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.149697    5784 scope.go:122] "RemoveContainer" containerID="e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"
Nov 03 12:32:39.160291 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.160273    5784 scope.go:122] "RemoveContainer" containerID="e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"
Nov 03 12:32:39.160594 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:39.160575    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95\": container with ID starting with e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95 not found: ID does not exist" containerID="e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"
Nov 03 12:32:39.160668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.160605    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95"} err="failed to get container status \"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95\": rpc error: code = NotFound desc = could not find container \"e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95\": container with ID starting with e03bfc6c36056019572e6772c568716a72287b7b11eb793608dff8ee1d688c95 not found: ID does not exist"
Nov 03 12:32:39.237939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.237910    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:39.302915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.302883    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:32:39.305851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.305824    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2daac7-fc4d4444c-scwtk"]
Nov 03 12:32:39.681897 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:39.681865    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74336dec-d18f-4043-a09e-a010607e47ab" path="/var/lib/kubelet/pods/74336dec-d18f-4043-a09e-a010607e47ab/volumes"
Nov 03 12:32:40.313487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:40.313452    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:44.262845 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:44.262782    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:44.789386 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:44.789351    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:44.944074 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:44.944043    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:45.104344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:45.104248    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:46.170547 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:46.170516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:46.442590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:46.442489    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:46.780089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:46.779987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:47.433366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:47.433333    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:48.192447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:48.192410    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:48.402231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:48.402198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:48.550891 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:48.550797    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:49.022007 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:49.021974    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:49.429807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:49.429770    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:49.595014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:49.594974    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:49.814932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:49.814839    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:52.797218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:52.797187    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:52.968347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:52.968307    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:53.585726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:53.585691    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:53.750456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:53.750421    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:53.916042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:53.916015    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:54.169846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:54.169760    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:54.333728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:54.333683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:54.477731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:54.477645    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:54.657605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:54.657571    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:54.913031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:54.912996    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:55.064269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:55.064236    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:56.472396 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:56.472350    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:32:57.375940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:57.375905    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:57.542742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:57.542704    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:59.356887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.356825    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 12:32:59.357465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.357446    5784 cpu_manager.go:414] "containerMap: removing container" podUID="74336dec-d18f-4043-a09e-a010607e47ab" containerName="maas-api"
Nov 03 12:32:59.357523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.357465    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74336dec-d18f-4043-a09e-a010607e47ab" containerName="maas-api"
Nov 03 12:32:59.357608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.357595    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="74336dec-d18f-4043-a09e-a010607e47ab" containerName="maas-api"
Nov 03 12:32:59.361405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.361369    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.416520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.416484    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzcbj\" (UniqueName: \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.416520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.416548    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.416807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.416696    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.416869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.416843    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.456115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.456080    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1250b3-serving-cert\""
Nov 03 12:32:59.459276 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.459253    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 12:32:59.517914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.517881    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzcbj\" (UniqueName: \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.518085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.517939    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.518085 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:59.518035    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-serving-cert: secret "maas-api-e2e-shared-a-w3-1250b3-serving-cert" not found
Nov 03 12:32:59.518085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.518063    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.518244 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:32:59.518088    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls podName:cc7fb1c3-dc45-416a-ae66-d14ef00867c9 nodeName:}" failed. No retries permitted until 2026-11-03 12:33:00.018073518 +0000 UTC m=+2486.460275815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" (UID: "cc7fb1c3-dc45-416a-ae66-d14ef00867c9") : secret "maas-api-e2e-shared-a-w3-1250b3-serving-cert" not found
Nov 03 12:32:59.518244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.518174    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.518700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.518678    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.520764 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.520728    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.571883 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.571851    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzcbj\" (UniqueName: \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:32:59.574744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.574660    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:59.727614 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.727585    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:32:59.883932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:32:59.883898    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:00.024648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:00.024559    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:33:00.027207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:00.027183    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:33:00.277604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:00.277518    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:33:00.644143 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:00.643980    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 12:33:00.645108 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:33:00.645082    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7fb1c3_dc45_416a_ae66_d14ef00867c9.slice/crio-2eaf2d18c6dd4dc2a37c98a8d7c9a14f5503bb004d82ecee51a7f6a2441fbe7f WatchSource:0}: Error finding container 2eaf2d18c6dd4dc2a37c98a8d7c9a14f5503bb004d82ecee51a7f6a2441fbe7f: Status 404 returned error can't find the container with id 2eaf2d18c6dd4dc2a37c98a8d7c9a14f5503bb004d82ecee51a7f6a2441fbe7f
Nov 03 12:33:01.427551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:01.427511    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" event={"ID":"cc7fb1c3-dc45-416a-ae66-d14ef00867c9","Type":"ContainerStarted","Data":"2eaf2d18c6dd4dc2a37c98a8d7c9a14f5503bb004d82ecee51a7f6a2441fbe7f"}
Nov 03 12:33:02.444972 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:02.444929    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" event={"ID":"cc7fb1c3-dc45-416a-ae66-d14ef00867c9","Type":"ContainerStarted","Data":"31c0ea44be23fcb6b002a202726cb65ad84685af1c0ca2efe969cb711c25d495"}
Nov 03 12:33:02.445489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:02.445172    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:33:02.551368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:02.551301    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" podStartSLOduration=2.695389884 podStartE2EDuration="3.551282926s" totalImagesPullingTime="855.893042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:02.54936343 +0000 UTC m=+2488.991565754" watchObservedRunningTime="2026-11-03 12:33:02.551282926 +0000 UTC m=+2488.993485233"
Nov 03 12:33:04.141160 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:04.141128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:06.631120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:06.631083    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:08.459618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:08.459590    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 12:33:08.561411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:08.560679    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:10.503012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:10.502975    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:10.657730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:10.657679    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:12.736399 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.735415    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 12:33:12.782712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.781833    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:12.812283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.812245    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 12:33:12.834601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.834516    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-ab27b8-serving-cert\""
Nov 03 12:33:12.884001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.883905    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:12.898785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.898724    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:12.899014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.898797    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdm4\" (UniqueName: \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:12.899014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.898912    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:12.899014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.898993    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.000056 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:12.999957    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.003301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.000998    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.003301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.002832    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.003301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.002875    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdm4\" (UniqueName: \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.004396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.003985    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.005521 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.005498    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.007223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.007198    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.038754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.038619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:13.073330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.073290    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdm4\" (UniqueName: \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\") pod \"maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.144884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.143672    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:13.193120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.193083    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:13.702411 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:33:13.702334    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b91d248_e578_4e1c_a954_d281e4c66775.slice/crio-cbab14a440d5c2c70891d769aed0b3e26261196a4e1b35d1b97ad8e2a58c13a7 WatchSource:0}: Error finding container cbab14a440d5c2c70891d769aed0b3e26261196a4e1b35d1b97ad8e2a58c13a7: Status 404 returned error can't find the container with id cbab14a440d5c2c70891d769aed0b3e26261196a4e1b35d1b97ad8e2a58c13a7
Nov 03 12:33:13.705555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.705533    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 12:33:13.759075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.752037    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:33:13.788948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.788908    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.806455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.806422    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:33:13.816039 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.816001    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.818237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.818210    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.819161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.818426    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.819367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.819348    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.819524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.819496    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.819748 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.819731    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.819914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.819899    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxs4c\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.820022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.820010    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.820118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.820105    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921593    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921635    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921668    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921716    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921744    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.921837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921794    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.922155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921853    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxs4c\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.922155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921884    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.922155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.921913    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.923493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.923098    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.924299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.924120    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.924299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.924262    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.924655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.924610    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.925041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.925019    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.925171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.925146    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.925171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.925162    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.976229 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.976150    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxs4c\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:13.976373 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:13.976338    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\") pod \"e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") " pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:14.124349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.124307    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:14.548580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.548544    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" event={"ID":"0b91d248-e578-4e1c-a954-d281e4c66775","Type":"ContainerStarted","Data":"cbab14a440d5c2c70891d769aed0b3e26261196a4e1b35d1b97ad8e2a58c13a7"}
Nov 03 12:33:14.559875 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:33:14.559841    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc878c14_eae2_4e23_b732_ffbff8a28bca.slice/crio-aa08e5067c62c7b064f2cdd161f64e672b4ac1f6900e4c01353c930afff4c7f7 WatchSource:0}: Error finding container aa08e5067c62c7b064f2cdd161f64e672b4ac1f6900e4c01353c930afff4c7f7: Status 404 returned error can't find the container with id aa08e5067c62c7b064f2cdd161f64e672b4ac1f6900e4c01353c930afff4c7f7
Nov 03 12:33:14.560030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.559982    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:33:14.562211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.562181    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:33:14.562311 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.562244    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:33:14.562311 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:14.562274    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:33:15.561414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.561360    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" event={"ID":"bc878c14-eae2-4e23-b732-ffbff8a28bca","Type":"ContainerStarted","Data":"684288b7f72de52788d8eaf7e39cdcd9b032ae7dc1331ee3389179dbd2c5aaaa"}
Nov 03 12:33:15.561795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.561426    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" event={"ID":"bc878c14-eae2-4e23-b732-ffbff8a28bca","Type":"ContainerStarted","Data":"aa08e5067c62c7b064f2cdd161f64e672b4ac1f6900e4c01353c930afff4c7f7"}
Nov 03 12:33:15.563530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.563505    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" event={"ID":"0b91d248-e578-4e1c-a954-d281e4c66775","Type":"ContainerStarted","Data":"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"}
Nov 03 12:33:15.563641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.563615    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:15.722226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.722157    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podStartSLOduration=2.7221354719999997 podStartE2EDuration="2.722135472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:15.714633732 +0000 UTC m=+2502.156836049" watchObservedRunningTime="2026-11-03 12:33:15.722135472 +0000 UTC m=+2502.164337779"
Nov 03 12:33:15.846931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:15.846808    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" podStartSLOduration=2.899583345 podStartE2EDuration="3.846783928s" totalImagesPullingTime="947.200583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:15.835708625 +0000 UTC m=+2502.277910952" watchObservedRunningTime="2026-11-03 12:33:15.846783928 +0000 UTC m=+2502.288986234"
Nov 03 12:33:16.137460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:16.125731    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:16.137460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:16.130499    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:16.137460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:16.130571    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:17.125783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:17.125741    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:17.126369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:17.126336    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:17.646403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:17.645274    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:18.128812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:18.128779    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:18.129460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:18.129417    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:19.136403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:19.126715    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:19.136403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:19.126774    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:20.141401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:20.139506    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:20.141401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:20.139573    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:21.125124 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:21.125086    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 12:33:21.125294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:21.125149    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 12:33:21.576256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:21.576223    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 12:33:22.129518 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:22.129479    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:22.129699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:22.129624    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:22.129982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:22.129965    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:22.130660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:22.130640    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:33:24.052715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.052685    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:33:24.056948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.056927    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.105485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.105454    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 12:33:24.105660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.105471    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 12:33:24.105660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.105479    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 12:33:24.105754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.105660    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rhbbs\""
Nov 03 12:33:24.156645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.156605    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:33:24.162504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162466    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.162650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162601    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.162693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162682    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5czkc\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.162798 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162773    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.162878 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162860    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.162984 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.162967    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.163054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.163031    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.264539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264496    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.264762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264563    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.264762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264594    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.264762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264681    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.264762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264761    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5czkc\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.265000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264794    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.265000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.264858    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.265346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.265322    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.267212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.267187    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.267345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.267330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.267437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.267332    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.267437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.267410    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.318625 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.318559    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.318795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.318777    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5czkc\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gdvvx\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.370552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.370524    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:24.792719 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.792683    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:33:24.828989 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:33:24.828960    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc87c1a_fdb0_4cd2_9184_d4b8724d57d4.slice/crio-6ae61030cd4ddf8c7b2ea2c685cc2f56bcdc466bd77ab5c506410fcc05fb0494 WatchSource:0}: Error finding container 6ae61030cd4ddf8c7b2ea2c685cc2f56bcdc466bd77ab5c506410fcc05fb0494: Status 404 returned error can't find the container with id 6ae61030cd4ddf8c7b2ea2c685cc2f56bcdc466bd77ab5c506410fcc05fb0494
Nov 03 12:33:24.831462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.831430    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:33:24.831544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:24.831493    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:33:25.640929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.640876    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" event={"ID":"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4","Type":"ContainerStarted","Data":"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"}
Nov 03 12:33:25.640929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.640929    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" event={"ID":"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4","Type":"ContainerStarted","Data":"6ae61030cd4ddf8c7b2ea2c685cc2f56bcdc466bd77ab5c506410fcc05fb0494"}
Nov 03 12:33:25.641434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.641039    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:25.642772 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.642744    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gdvvx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:33:25.642931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.642794    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:33:25.749946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:25.749890    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" podStartSLOduration=1.749875056 podStartE2EDuration="1.749875056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:33:25.748166469 +0000 UTC m=+2512.190368773" watchObservedRunningTime="2026-11-03 12:33:25.749875056 +0000 UTC m=+2512.192077361"
Nov 03 12:33:26.497490 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:33:26.497411    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:33:26.548018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:26.547981    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:26.648450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:26.648419    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gdvvx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:33:26.648851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:26.648471    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:33:29.648769 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:29.648728    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:33:30.823360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:30.823322    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:30.973115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:30.973077    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:31.113212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:31.113133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:35.439391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:35.439351    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:39.771252 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:39.771213    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:39.951870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:39.951827    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:40.227816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:40.227780    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:44.360261 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:44.360184    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:44.517033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:44.516999    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:44.666568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:44.666529    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:49.017507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:49.017467    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:53.416706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:53.416666    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:54.482267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:54.482228    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:33:56.472512 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:33:56.472476    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:33:57.916599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:33:57.916559    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:01.155394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:01.155348    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:01.636370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:01.636337    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:02.647212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:02.647171    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:02.799772 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:02.799732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:02.947801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:02.947696    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:03.731074 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.731035    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:34:03.736072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.736052    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.784119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.784093    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bbca30-openshift-default-dockercfg-xx7jx\""
Nov 03 12:34:03.845887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.845850    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.845926    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.845954    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.845984    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.846074    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.846107    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcjrh\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.846206    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.846272    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.846494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.846312    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.922101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.922068    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:34:03.947465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947430    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947517    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947549    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947583    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947687    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947723    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcjrh\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.947833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947818    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947859    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947889    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.947928    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.948309    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948569 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.948542    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.948672    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.948845 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.948824    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.950694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.950668    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:03.951123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:03.951100    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:04.000274 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.000207    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:04.000410 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.000357    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcjrh\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:04.052290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.052260    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:04.471888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.471853    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:34:04.474087 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:34:04.474053    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6919e792_d71d_4630_8ff3_9aadf36a921d.slice/crio-e68e6649df166391ab4957a65ca4df7c1ce940f8c7786a85a71810a2bcdceb18 WatchSource:0}: Error finding container e68e6649df166391ab4957a65ca4df7c1ce940f8c7786a85a71810a2bcdceb18: Status 404 returned error can't find the container with id e68e6649df166391ab4957a65ca4df7c1ce940f8c7786a85a71810a2bcdceb18
Nov 03 12:34:04.476903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.476857    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:04.477019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.476950    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:04.477019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.477001    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:04.924271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.924238    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" event={"ID":"6919e792-d71d-4630-8ff3-9aadf36a921d","Type":"ContainerStarted","Data":"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"}
Nov 03 12:34:04.924781 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:04.924290    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" event={"ID":"6919e792-d71d-4630-8ff3-9aadf36a921d","Type":"ContainerStarted","Data":"e68e6649df166391ab4957a65ca4df7c1ce940f8c7786a85a71810a2bcdceb18"}
Nov 03 12:34:05.033955 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:05.033895    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podStartSLOduration=2.03387811 podStartE2EDuration="2.03387811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:05.031532486 +0000 UTC m=+2551.473734812" watchObservedRunningTime="2026-11-03 12:34:05.03387811 +0000 UTC m=+2551.476080415"
Nov 03 12:34:05.053254 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:05.053217    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:05.054689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:05.054656    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:05.054855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:05.054717    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:06.052947 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:06.052907    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:06.053361 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:06.052963    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:07.053484 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:07.053450    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:07.053899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:07.053519    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:07.739404 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:07.738893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:07.904704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:07.904667    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:08.052888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:08.052806    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:08.052888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:08.052861    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:08.073884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:08.073856    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:09.053367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:09.053328    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:09.053568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:09.053414    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:10.053366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:10.053328    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 03 12:34:10.053762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:10.053397    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 03 12:34:10.886119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:10.886089    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:11.057269 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:11.057237    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:11.057678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:11.057627    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:11.058299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:11.058280    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:34:11.071677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:11.071645    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:12.650658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:12.650625    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:14.956704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:14.956667    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:15.428629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:15.428594    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:16.196321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:16.196285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:16.425556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:16.425477    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:17.466051 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:17.466012    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:18.556579 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:18.556541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:20.360898 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:20.360861    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:20.520577 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:20.520539    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:22.294686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:22.294656    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:23.422296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:23.422262    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:23.586494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:23.586462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:26.473334 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:34:26.473295    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:34:26.721289 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:26.721255    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:31.217310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:31.217276    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:36.042971 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:36.042935    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:36.752419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:36.752349    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:37.245982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:37.245947    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:37.817455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:37.817416    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:37.989024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:37.988967    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:39.328201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:39.328167    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:39.488025 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:39.487989    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:41.033041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:41.033005    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:42.587651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:42.587619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:42.754105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:42.754056    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:42.911253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:42.911213    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:44.988661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:44.988628    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:45.858726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:45.858689    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:46.097006 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:46.096976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:46.268075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:46.268020    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:47.463856 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:47.463819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:49.087287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.087252    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:34:49.091671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.091649    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162149    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162209    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162310    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdqzc\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162577 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162553    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162594    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162687    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162722    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162840 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162824    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.162943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.162922    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.198211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.198177    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:34:49.264163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.264124    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265223    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265516    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265554    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265608    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265774 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265673    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265785    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265812    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.265937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265842    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdqzc\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.266294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.265162    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.266938 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.266916    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.267370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.267133    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.267581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.267320    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.267581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.267402    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.269122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.269097    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.269991 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.269970    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.318917 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.318866    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.319084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.318957    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdqzc\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\") pod \"e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") " pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.405951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.405912    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:49.830597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.830563    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:34:49.831751 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:34:49.831721    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cfa200_3092_4931_a1b7_2d2827071487.slice/crio-9267137be455b5f1a637af0d7fc66d1937c24e9a02e7b3e78f4c8ef4168ce9b0 WatchSource:0}: Error finding container 9267137be455b5f1a637af0d7fc66d1937c24e9a02e7b3e78f4c8ef4168ce9b0: Status 404 returned error can't find the container with id 9267137be455b5f1a637af0d7fc66d1937c24e9a02e7b3e78f4c8ef4168ce9b0
Nov 03 12:34:49.835188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.835144    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:49.835270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.835233    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:49.835309 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:49.835277    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:34:50.305822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.305780    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" event={"ID":"a2cfa200-3092-4931-a1b7-2d2827071487","Type":"ContainerStarted","Data":"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"}
Nov 03 12:34:50.306314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.305844    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" event={"ID":"a2cfa200-3092-4931-a1b7-2d2827071487","Type":"ContainerStarted","Data":"9267137be455b5f1a637af0d7fc66d1937c24e9a02e7b3e78f4c8ef4168ce9b0"}
Nov 03 12:34:50.406335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.406304    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:50.407644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.407616    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:50.407783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.407681    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:50.413946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.413897    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podStartSLOduration=1.4138844609999999 podStartE2EDuration="1.413884461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:34:50.412688859 +0000 UTC m=+2596.854891163" watchObservedRunningTime="2026-11-03 12:34:50.413884461 +0000 UTC m=+2596.856086765"
Nov 03 12:34:50.746208 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:50.746161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:51.261730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:51.261691    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:51.406866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:51.406834    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:51.407250 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:51.406886    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:52.406452 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:52.406368    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:52.406658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:52.406505    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:52.488339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:52.488307    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:53.234848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:53.234801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:53.397456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:53.397413    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:53.406410 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:53.406359    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:53.406565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:53.406452    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:53.825297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:53.825250    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:54.344492 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:54.344443    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:54.406806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:54.406743    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:54.407067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:54.406867    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:54.493255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:54.493213    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:54.852793 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:54.852749    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:55.419403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:55.414795    5784 patch_prober.go:29] interesting pod/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 12:34:55.419403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:55.414868    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 12:34:55.546657 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:55.546569    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:56.411408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:56.411354    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:56.411854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:56.411694    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:56.412519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:56.412500    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:34:56.476583 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:34:56.476546    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:34:57.651416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:57.651360    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:57.807298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:57.807263    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:57.966556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:57.966470    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:34:59.431829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:34:59.431774    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:01.083929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:01.083893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:05.162957 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.162923    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 12:35:05.167092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.167071    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.205576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.205544    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmlr\" (UniqueName: \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.205746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.205586    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.205746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.205728    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.205827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.205766    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.262264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.262224    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-8570a0-serving-cert\""
Nov 03 12:35:05.267542 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.267508    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 12:35:05.306909 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.306868    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmlr\" (UniqueName: \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.307084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.306936    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.307084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.307048    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.307191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.307093    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.307326 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:05.307309    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-serving-cert: secret "maas-api-e2e-shared-b-w3-8570a0-serving-cert" not found
Nov 03 12:35:05.307411 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:05.307388    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls podName:407e4d6e-0b50-4652-b230-13407245d514 nodeName:}" failed. No retries permitted until 2026-11-03 12:35:05.807353185 +0000 UTC m=+2612.249555485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls") pod "maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" (UID: "407e4d6e-0b50-4652-b230-13407245d514") : secret "maas-api-e2e-shared-b-w3-8570a0-serving-cert" not found
Nov 03 12:35:05.307851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.307820    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.309945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.309910    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.360604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.360568    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmlr\" (UniqueName: \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.365307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.365276    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:05.522455 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.522348    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:05.674238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.674205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:05.822442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.822348    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:05.825115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:05.825090    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:06.081088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:06.080996    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:06.451520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:06.451487    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 12:35:06.453757 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:35:06.453715    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407e4d6e_0b50_4652_b230_13407245d514.slice/crio-78f4c9c6555ba9424658efdaca8ddcf718fb7458bb928399f1b177e576b1399e WatchSource:0}: Error finding container 78f4c9c6555ba9424658efdaca8ddcf718fb7458bb928399f1b177e576b1399e: Status 404 returned error can't find the container with id 78f4c9c6555ba9424658efdaca8ddcf718fb7458bb928399f1b177e576b1399e
Nov 03 12:35:06.506356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:06.506314    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" event={"ID":"407e4d6e-0b50-4652-b230-13407245d514","Type":"ContainerStarted","Data":"78f4c9c6555ba9424658efdaca8ddcf718fb7458bb928399f1b177e576b1399e"}
Nov 03 12:35:08.524934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:08.524891    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" event={"ID":"407e4d6e-0b50-4652-b230-13407245d514","Type":"ContainerStarted","Data":"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"}
Nov 03 12:35:08.525355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:08.524968    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:08.633977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:08.633921    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" podStartSLOduration=2.634453911 podStartE2EDuration="3.633905988s" totalImagesPullingTime="999.452077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:08.630673048 +0000 UTC m=+2615.072875354" watchObservedRunningTime="2026-11-03 12:35:08.633905988 +0000 UTC m=+2615.076108292"
Nov 03 12:35:10.349121 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:10.349088    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:14.539419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:14.539319    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 12:35:15.152358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:15.152318    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:18.865695 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:18.865659    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:35:18.866188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:18.865929    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" podUID="8536dde7-8075-4743-bac5-050277e0b7cb" containerName="maas-api" containerID="cri-o://f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935" gracePeriod=30
Nov 03 12:35:18.990313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:18.990278    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:19.211604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.211574    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:35:19.342367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.341396    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\") pod \"8536dde7-8075-4743-bac5-050277e0b7cb\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") "
Nov 03 12:35:19.342367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.341470    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") pod \"8536dde7-8075-4743-bac5-050277e0b7cb\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") "
Nov 03 12:35:19.342367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.341531    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\") pod \"8536dde7-8075-4743-bac5-050277e0b7cb\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") "
Nov 03 12:35:19.342367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.341587    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\" (UniqueName: \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\") pod \"8536dde7-8075-4743-bac5-050277e0b7cb\" (UID: \"8536dde7-8075-4743-bac5-050277e0b7cb\") "
Nov 03 12:35:19.342367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.342223    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca" pod "8536dde7-8075-4743-bac5-050277e0b7cb" (UID: "8536dde7-8075-4743-bac5-050277e0b7cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:19.344620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.344566    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls" pod "8536dde7-8075-4743-bac5-050277e0b7cb" (UID: "8536dde7-8075-4743-bac5-050277e0b7cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:35:19.345866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.345829    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs" pod "8536dde7-8075-4743-bac5-050277e0b7cb" (UID: "8536dde7-8075-4743-bac5-050277e0b7cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:35:19.350341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.350244    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt" pod "8536dde7-8075-4743-bac5-050277e0b7cb" (UID: "8536dde7-8075-4743-bac5-050277e0b7cb"). InnerVolumeSpecName "kube-api-access-lxnxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:19.443168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.443040    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8536dde7-8075-4743-bac5-050277e0b7cb-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:19.443168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.443073    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:19.443168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.443082    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8536dde7-8075-4743-bac5-050277e0b7cb-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:19.443168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.443090    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxnxt\" (UniqueName: \"kubernetes.io/projected/8536dde7-8075-4743-bac5-050277e0b7cb-kube-api-access-lxnxt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:19.609973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.609937    5784 generic.go:356] "Generic (PLEG): container finished" podID="8536dde7-8075-4743-bac5-050277e0b7cb" containerID="f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935" exitCode=0
Nov 03 12:35:19.610155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.609984    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"
Nov 03 12:35:19.610155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.610025    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" event={"ID":"8536dde7-8075-4743-bac5-050277e0b7cb","Type":"ContainerDied","Data":"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"}
Nov 03 12:35:19.610155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.610079    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx" event={"ID":"8536dde7-8075-4743-bac5-050277e0b7cb","Type":"ContainerDied","Data":"a18fd96a557df00eb40c0cff38d118bbaf80c1996d050cb0a66d180c5b1dc2bc"}
Nov 03 12:35:19.610155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.610102    5784 scope.go:122] "RemoveContainer" containerID="f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"
Nov 03 12:35:19.621341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.621322    5784 scope.go:122] "RemoveContainer" containerID="f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"
Nov 03 12:35:19.621615 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:19.621596    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935\": container with ID starting with f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935 not found: ID does not exist" containerID="f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"
Nov 03 12:35:19.621667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.621623    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935"} err="failed to get container status \"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935\": rpc error: code = NotFound desc = could not find container \"f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935\": container with ID starting with f4d628f6298787c6bcaab1044da4d76e6c9f6a0dcd2169c716a5801724e95935 not found: ID does not exist"
Nov 03 12:35:19.773537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.773435    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:35:19.784357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:19.784137    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-eb4a0f-8465dd5f8b-bqvlx"]
Nov 03 12:35:20.300566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.300532    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 12:35:20.301113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.301100    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8536dde7-8075-4743-bac5-050277e0b7cb" containerName="maas-api"
Nov 03 12:35:20.301162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.301114    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8536dde7-8075-4743-bac5-050277e0b7cb" containerName="maas-api"
Nov 03 12:35:20.301225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.301216    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8536dde7-8075-4743-bac5-050277e0b7cb" containerName="maas-api"
Nov 03 12:35:20.305825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.305806    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.353443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.353396    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqkt\" (UniqueName: \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.353756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.353734    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.353934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.353917    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.354171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.354153    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.355640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.355581    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-743d9b-serving-cert\""
Nov 03 12:35:20.378534 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.378500    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 12:35:20.455924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.455881    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.456123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.456016    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqkt\" (UniqueName: \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.456123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.456075    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.456270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.456133    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.456353 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:20.456287    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-serving-cert: secret "maas-api-e2e-shared-b-w0-743d9b-serving-cert" not found
Nov 03 12:35:20.456433 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:20.456366    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls podName:d9415789-a9c5-4ae5-b47f-dc943676cc5f nodeName:}" failed. No retries permitted until 2026-11-03 12:35:20.956343543 +0000 UTC m=+2627.398545846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" (UID: "d9415789-a9c5-4ae5-b47f-dc943676cc5f") : secret "maas-api-e2e-shared-b-w0-743d9b-serving-cert" not found
Nov 03 12:35:20.457543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.457513    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.459417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.459371    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.509182 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.509150    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqkt\" (UniqueName: \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.514723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.514686    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:20.663810 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.663778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:20.816780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.816741    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:20.965108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.965019    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:20.967746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:20.967715    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:21.220214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:21.220125    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:21.594879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:21.594848    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 12:35:21.595206 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:35:21.594974    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9415789_a9c5_4ae5_b47f_dc943676cc5f.slice/crio-ea9e60e9a2e9a0da4f9b4e3efd26844a5c488dcfb82a62dd2470a0bf02ab52e0 WatchSource:0}: Error finding container ea9e60e9a2e9a0da4f9b4e3efd26844a5c488dcfb82a62dd2470a0bf02ab52e0: Status 404 returned error can't find the container with id ea9e60e9a2e9a0da4f9b4e3efd26844a5c488dcfb82a62dd2470a0bf02ab52e0
Nov 03 12:35:21.630442 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:21.630391    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" event={"ID":"d9415789-a9c5-4ae5-b47f-dc943676cc5f","Type":"ContainerStarted","Data":"ea9e60e9a2e9a0da4f9b4e3efd26844a5c488dcfb82a62dd2470a0bf02ab52e0"}
Nov 03 12:35:21.681900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:21.681864    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8536dde7-8075-4743-bac5-050277e0b7cb" path="/var/lib/kubelet/pods/8536dde7-8075-4743-bac5-050277e0b7cb/volumes"
Nov 03 12:35:22.054536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:22.054506    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:22.218123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:22.218073    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:22.638729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:22.638697    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" event={"ID":"d9415789-a9c5-4ae5-b47f-dc943676cc5f","Type":"ContainerStarted","Data":"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"}
Nov 03 12:35:22.639149 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:22.638808    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:22.760475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:22.760318    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" podStartSLOduration=1.870530197 podStartE2EDuration="2.760304859s" totalImagesPullingTime="889.774662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:35:22.758018441 +0000 UTC m=+2629.200220764" watchObservedRunningTime="2026-11-03 12:35:22.760304859 +0000 UTC m=+2629.202507197"
Nov 03 12:35:23.482880 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:23.482848    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:23.954189 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:23.954160    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:25.517069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:25.517038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:26.472749 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:26.472702    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:35:28.652093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:28.652060    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 12:35:29.603530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:29.603487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:30.219628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:30.219573    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:32.190089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:32.190051    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 12:35:32.190581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:32.190315    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy" containerID="cri-o://4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a" gracePeriod=30
Nov 03 12:35:34.865629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:34.865539    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:35.405215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:35.405179    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:36.752181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:36.752150    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:36.900231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:36.900197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:37.471133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.471064    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 12:35:37.587637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587603    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.587824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587716    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.587824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587738    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.587824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587793    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.587824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587820    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.588053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587868    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:37.588053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587884    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.588053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587910    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.588053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.587954    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.588053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588021    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:37.588310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588058    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\") pod \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\" (UID: \"9b4bfd5e-7bef-49d0-854d-3c550fc77705\") "
Nov 03 12:35:37.588310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588180    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:37.588439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588315    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:37.588502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588477    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:37.588771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588750    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.588865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588775    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.588865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588791    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.588865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588806    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.588865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.588821    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.590145 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.590122    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "kube-api-access-l8fv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:37.590234 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.590151    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:37.590436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.590411    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:37.590436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.590419    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo" pod "9b4bfd5e-7bef-49d0-854d-3c550fc77705" (UID: "9b4bfd5e-7bef-49d0-854d-3c550fc77705"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:37.690033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.689967    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.690033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.689990    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.690033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.690000    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fv2\" (UniqueName: \"kubernetes.io/projected/9b4bfd5e-7bef-49d0-854d-3c550fc77705-kube-api-access-l8fv2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.690033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.690020    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4bfd5e-7bef-49d0-854d-3c550fc77705-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:37.760371 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.760277    5784 generic.go:356] "Generic (PLEG): container finished" podID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerID="4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a" exitCode=0
Nov 03 12:35:37.760997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.760969    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerDied","Data":"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"}
Nov 03 12:35:37.761163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.761148    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f" event={"ID":"9b4bfd5e-7bef-49d0-854d-3c550fc77705","Type":"ContainerDied","Data":"e0a4d2bef09aaedb77833e47c397bdef127a4ae5c02bbb038c262c1a9ac285a9"}
Nov 03 12:35:37.761270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.761259    5784 scope.go:122] "RemoveContainer" containerID="4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"
Nov 03 12:35:37.761550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.761531    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"
Nov 03 12:35:37.829893 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.829453    5784 scope.go:122] "RemoveContainer" containerID="3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"
Nov 03 12:35:37.866024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.861534    5784 scope.go:122] "RemoveContainer" containerID="4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"
Nov 03 12:35:37.866778 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:37.866399    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a\": container with ID starting with 4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a not found: ID does not exist" containerID="4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"
Nov 03 12:35:37.866778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.866466    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a"} err="failed to get container status \"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a\": rpc error: code = NotFound desc = could not find container \"4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a\": container with ID starting with 4587ed91627ae46b218a1e728b668894834a0a3b684d0fb1ed9a49cee7716c3a not found: ID does not exist"
Nov 03 12:35:37.866778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.866494    5784 scope.go:122] "RemoveContainer" containerID="3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"
Nov 03 12:35:37.867235 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:37.867144    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed\": container with ID starting with 3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed not found: ID does not exist" containerID="3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"
Nov 03 12:35:37.867235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.867174    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed"} err="failed to get container status \"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed\": rpc error: code = NotFound desc = could not find container \"3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed\": container with ID starting with 3b74c581c3218ed8bf58bd17d045de427dfe0d68a566fe1be5a134c10a61b5ed not found: ID does not exist"
Nov 03 12:35:37.930491 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.930454    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 12:35:37.937945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:37.937917    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-6584f"]
Nov 03 12:35:39.009200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:39.009163    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:39.682757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:39.682714    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" path="/var/lib/kubelet/pods/9b4bfd5e-7bef-49d0-854d-3c550fc77705/volumes"
Nov 03 12:35:40.032822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:40.032724    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:40.184724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:40.184683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:40.634019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:40.633984    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:40.827935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:40.827901    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:41.014519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:41.014449    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:42.768152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:42.768116    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:43.250711 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:43.250673    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:35:43.251019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:43.250983    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy" containerID="cri-o://b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf" gracePeriod=30
Nov 03 12:35:43.258144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:43.258113    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:35:43.258408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:43.258353    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy" containerID="cri-o://5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639" gracePeriod=30
Nov 03 12:35:43.274646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:43.274613    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:45.796937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:45.796908    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:48.556418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.556361    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:35:48.586943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.586920    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:35:48.624139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624097    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624194    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624220    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624275    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624330 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624320    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624357    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624397    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624439    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624474    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624513    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624587    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624662    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624676    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624702    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624740    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624762    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624804    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.624920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624883    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.625272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624922    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\") pod \"a2cfa200-3092-4931-a1b7-2d2827071487\" (UID: \"a2cfa200-3092-4931-a1b7-2d2827071487\") "
Nov 03 12:35:48.625272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624958    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\") pod \"6919e792-d71d-4630-8ff3-9aadf36a921d\" (UID: \"6919e792-d71d-4630-8ff3-9aadf36a921d\") "
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625550    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625574    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624952    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.624992    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625216    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625601    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625890    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:35:48.625948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.625907    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.626494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.626020    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.626494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.626065    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.627097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.627061    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "kube-api-access-dcjrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.627509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.627482    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.628270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.628243    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.628712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.628678    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "kube-api-access-zdqzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.628801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.628766    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:35:48.628927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.628913    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:35:48.629005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.628980    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo" pod "a2cfa200-3092-4931-a1b7-2d2827071487" (UID: "a2cfa200-3092-4931-a1b7-2d2827071487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.629042 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.629006    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo" pod "6919e792-d71d-4630-8ff3-9aadf36a921d" (UID: "6919e792-d71d-4630-8ff3-9aadf36a921d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:35:48.727417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727356    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2cfa200-3092-4931-a1b7-2d2827071487-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727416    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727434    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdqzc\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-kube-api-access-zdqzc\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727465    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727489    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727506    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727521    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6919e792-d71d-4630-8ff3-9aadf36a921d-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727537    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2cfa200-3092-4931-a1b7-2d2827071487-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727551    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727566    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727581    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcjrh\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-kube-api-access-dcjrh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727596    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2cfa200-3092-4931-a1b7-2d2827071487-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727610    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6919e792-d71d-4630-8ff3-9aadf36a921d-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727624    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727640    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2cfa200-3092-4931-a1b7-2d2827071487-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.727688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.727657    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6919e792-d71d-4630-8ff3-9aadf36a921d-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:35:48.857370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.857282    5784 generic.go:356] "Generic (PLEG): container finished" podID="a2cfa200-3092-4931-a1b7-2d2827071487" containerID="5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639" exitCode=0
Nov 03 12:35:48.857370 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.857337    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"
Nov 03 12:35:48.857610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.857538    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" event={"ID":"a2cfa200-3092-4931-a1b7-2d2827071487","Type":"ContainerDied","Data":"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"}
Nov 03 12:35:48.857610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.857587    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp" event={"ID":"a2cfa200-3092-4931-a1b7-2d2827071487","Type":"ContainerDied","Data":"9267137be455b5f1a637af0d7fc66d1937c24e9a02e7b3e78f4c8ef4168ce9b0"}
Nov 03 12:35:48.857715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.857615    5784 scope.go:122] "RemoveContainer" containerID="5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"
Nov 03 12:35:48.860565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.860535    5784 generic.go:356] "Generic (PLEG): container finished" podID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerID="b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf" exitCode=0
Nov 03 12:35:48.860723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.860617    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" event={"ID":"6919e792-d71d-4630-8ff3-9aadf36a921d","Type":"ContainerDied","Data":"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"}
Nov 03 12:35:48.860723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.860669    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd" event={"ID":"6919e792-d71d-4630-8ff3-9aadf36a921d","Type":"ContainerDied","Data":"e68e6649df166391ab4957a65ca4df7c1ce940f8c7786a85a71810a2bcdceb18"}
Nov 03 12:35:48.860843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.860749    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"
Nov 03 12:35:48.873337 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.873219    5784 scope.go:122] "RemoveContainer" containerID="5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"
Nov 03 12:35:48.875090 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:48.875068    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639\": container with ID starting with 5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639 not found: ID does not exist" containerID="5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"
Nov 03 12:35:48.875211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.875097    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639"} err="failed to get container status \"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639\": rpc error: code = NotFound desc = could not find container \"5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639\": container with ID starting with 5e4bfd30b77700113d33ea91c29b08e56cf36a5812d911e416df4cbf9c535639 not found: ID does not exist"
Nov 03 12:35:48.875211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.875121    5784 scope.go:122] "RemoveContainer" containerID="b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"
Nov 03 12:35:48.884585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.884566    5784 scope.go:122] "RemoveContainer" containerID="b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"
Nov 03 12:35:48.884842 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:48.884820    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf\": container with ID starting with b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf not found: ID does not exist" containerID="b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"
Nov 03 12:35:48.884937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:48.884846    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf"} err="failed to get container status \"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf\": rpc error: code = NotFound desc = could not find container \"b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf\": container with ID starting with b9b17b682d14dd41faab23a205afeff9d08657b0dc93d0c26b83b546149b85cf not found: ID does not exist"
Nov 03 12:35:49.010240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.010206    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:35:49.015140 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.015114    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-gsnxd"]
Nov 03 12:35:49.116853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.116756    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:35:49.123260 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.123228    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bbca30-openshift-default-6577cd76cc-nhmqp"]
Nov 03 12:35:49.701608 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.701572    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" path="/var/lib/kubelet/pods/6919e792-d71d-4630-8ff3-9aadf36a921d/volumes"
Nov 03 12:35:49.702061 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:49.702045    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" path="/var/lib/kubelet/pods/a2cfa200-3092-4931-a1b7-2d2827071487/volumes"
Nov 03 12:35:50.173701 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:50.173659    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:52.422391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:52.422354    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:53.044312 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:53.044275    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:53.352006 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:53.351900    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:54.603595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:54.603556    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:55.186419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:55.186354    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:55.325814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:55.325782    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:56.472362 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:35:56.472315    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:35:57.489772 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:57.489736    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:57.675031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:57.674978    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:57.844038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:57.843947    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:35:59.500986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:35:59.500948    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:00.632181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:00.632146    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:00.777418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:00.777390    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:03.967454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.967417    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:36:03.968227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968204    5784 cpu_manager.go:414] "containerMap: removing container" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:36:03.968278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968233    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:36:03.968278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968258    5784 cpu_manager.go:414] "containerMap: removing container" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy"
Nov 03 12:36:03.968278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968263    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy"
Nov 03 12:36:03.968396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968284    5784 cpu_manager.go:414] "containerMap: removing container" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:36:03.968396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968288    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:36:03.968396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968297    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy"
Nov 03 12:36:03.968396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968302    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy"
Nov 03 12:36:03.968545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968412    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:36:03.968545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968423    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6919e792-d71d-4630-8ff3-9aadf36a921d" containerName="istio-proxy"
Nov 03 12:36:03.968545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.968444    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cfa200-3092-4931-a1b7-2d2827071487" containerName="istio-proxy"
Nov 03 12:36:03.972115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:03.972098    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.023968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.023935    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:36:04.065436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.065404    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-3176d9-serving-cert\""
Nov 03 12:36:04.134528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.134497    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82md\" (UniqueName: \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.134699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.134557    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.134699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.134677    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.134796 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.134779    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.173452 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.173372    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:04.236512 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.236412    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.236691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.236562    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.236771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.236706    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82md\" (UniqueName: \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.236829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.236787    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.236967 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:36:04.236950    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-serving-cert: secret "maas-api-e2e-praxis-b-3176d9-serving-cert" not found
Nov 03 12:36:04.237167 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:36:04.237128    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls podName:52af2b97-eb5e-4558-a649-5571fa899d02 nodeName:}" failed. No retries permitted until 2026-11-03 12:36:04.737107559 +0000 UTC m=+2671.179309841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls") pod "maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" (UID: "52af2b97-eb5e-4558-a649-5571fa899d02") : secret "maas-api-e2e-praxis-b-3176d9-serving-cert" not found
Nov 03 12:36:04.237428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.237364    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.240011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.239985    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.289886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.289857    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82md\" (UniqueName: \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.323198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.323168    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:04.475808 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.475775    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:04.741645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.741616    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.744084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.744051    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") pod \"maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:04.886091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:04.886064    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:05.255009 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:05.254977    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:36:05.256836 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:36:05.256798    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52af2b97_eb5e_4558_a649_5571fa899d02.slice/crio-6c78ba00bcaa7985310be68354d0018fba697f41cd8232c81a053207170fcde6 WatchSource:0}: Error finding container 6c78ba00bcaa7985310be68354d0018fba697f41cd8232c81a053207170fcde6: Status 404 returned error can't find the container with id 6c78ba00bcaa7985310be68354d0018fba697f41cd8232c81a053207170fcde6
Nov 03 12:36:05.984859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:05.984823    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" event={"ID":"52af2b97-eb5e-4558-a649-5571fa899d02","Type":"ContainerStarted","Data":"6c78ba00bcaa7985310be68354d0018fba697f41cd8232c81a053207170fcde6"}
Nov 03 12:36:06.993402 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:06.993343    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" event={"ID":"52af2b97-eb5e-4558-a649-5571fa899d02","Type":"ContainerStarted","Data":"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"}
Nov 03 12:36:06.993848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:06.993445    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:07.101068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:07.101012    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" podStartSLOduration=3.196148995 podStartE2EDuration="4.100996184s" totalImagesPullingTime="904.847189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:36:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:36:07.100027601 +0000 UTC m=+2673.542229925" watchObservedRunningTime="2026-11-03 12:36:07.100996184 +0000 UTC m=+2673.543198490"
Nov 03 12:36:08.798642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:08.798603    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:12.326420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:12.326360    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:13.006892 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:13.006862    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:36:13.204623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:13.204589    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:17.640943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:17.640909    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:20.215076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:20.215045    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:22.347564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:22.347526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:22.518544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:22.518498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:24.565625 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:24.565586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:26.473016 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:36:26.472976    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:36:26.903491 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:26.903455    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:31.328691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:31.328642    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:33.370948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:33.370901    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:34.183205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:34.183176    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:36:34.195717 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:34.195692    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:36:36.016716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:36.016673    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:40.824171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:40.824134    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:45.404089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:45.404038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:49.858985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:49.858954    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:54.502580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:54.502541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:54.665142 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:54.665099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:55.621417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:55.621313    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:36:56.477725 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:36:56.477684    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:36:58.934937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:36:58.934906    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:03.013067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.013032    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:37:03.013962 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.013933    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b4bfd5e-7bef-49d0-854d-3c550fc77705" containerName="istio-proxy"
Nov 03 12:37:03.017297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.017280    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.070817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.070786    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:37:03.112182 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.112146    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-561bd4-serving-cert\""
Nov 03 12:37:03.169329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.169277    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc8s4\" (UniqueName: \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.169728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.169695    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.169887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.169814    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.169887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.169847    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.228434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.228402    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:03.271464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.271346    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc8s4\" (UniqueName: \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.271688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.271523    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.271688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.271638    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.271688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.271668    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.272271 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:37:03.272058    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-serving-cert: secret "maas-api-e2e-autores-561bd4-serving-cert" not found
Nov 03 12:37:03.272271 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:37:03.272151    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls podName:de5a1383-a832-4700-8d73-1f88e294d5a5 nodeName:}" failed. No retries permitted until 2026-11-03 12:37:03.772130341 +0000 UTC m=+2730.214332638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls") pod "maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" (UID: "de5a1383-a832-4700-8d73-1f88e294d5a5") : secret "maas-api-e2e-autores-561bd4-serving-cert" not found
Nov 03 12:37:03.274425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.272790    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.281136 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.281103    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.325231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.325191    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc8s4\" (UniqueName: \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.381635 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.381599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:03.609263 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.609173    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:03.778975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.778940    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.781577 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.781548    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") pod \"maas-api-e2e-autores-561bd4-5c996f95d4-thwlr\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:03.931639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:03.931593    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:04.302486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:04.302454    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:37:04.303401 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:37:04.303339    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a1383_a832_4700_8d73_1f88e294d5a5.slice/crio-914990b9143f888525aa3fb48d17c670eeee03b405194462c3c849b53c788ec8 WatchSource:0}: Error finding container 914990b9143f888525aa3fb48d17c670eeee03b405194462c3c849b53c788ec8: Status 404 returned error can't find the container with id 914990b9143f888525aa3fb48d17c670eeee03b405194462c3c849b53c788ec8
Nov 03 12:37:04.435584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:04.435550    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" event={"ID":"de5a1383-a832-4700-8d73-1f88e294d5a5","Type":"ContainerStarted","Data":"914990b9143f888525aa3fb48d17c670eeee03b405194462c3c849b53c788ec8"}
Nov 03 12:37:05.443747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:05.443651    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" event={"ID":"de5a1383-a832-4700-8d73-1f88e294d5a5","Type":"ContainerStarted","Data":"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"}
Nov 03 12:37:05.444136 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:05.443744    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:05.550725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:05.550671    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" podStartSLOduration=2.673851889 podStartE2EDuration="3.550656522s" totalImagesPullingTime="876.804633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:37:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:37:05.550233608 +0000 UTC m=+2731.992435916" watchObservedRunningTime="2026-11-03 12:37:05.550656522 +0000 UTC m=+2731.992858826"
Nov 03 12:37:07.928414 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:07.928365    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:11.458795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:11.458759    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:37:12.422103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:12.422046    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:16.852223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:16.852180    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:20.923093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:20.922987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:21.622298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:21.622260    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:24.729863 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:24.729819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:26.110129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:26.110093    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:26.472291 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:37:26.472258    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:37:30.490589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:30.490560    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:34.850283 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:34.850240    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:35.212013 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:35.211978    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:39.250240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:39.250192    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:43.634738 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:43.634706    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:48.025318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:48.025279    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:52.401075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:52.401032    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:55.834345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:55.834304    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:37:56.476223 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:37:56.476172    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:37:56.755846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:37:56.755756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:01.236395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:01.236356    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:05.155449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:05.155407    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:05.610651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:05.610555    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:10.052662 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:10.052624    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:14.438181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:14.438099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:15.196848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:15.196814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:17.078541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:17.078505    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:38:17.079060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:17.078773    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy" containerID="cri-o://6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e" gracePeriod=30
Nov 03 12:38:19.092934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:19.092897    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:22.347648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.347583    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:38:22.463093 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463058    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463141    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463193    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463235    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463286 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463279    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463548 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463311    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463548 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463424    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463548 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463457    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463548 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463475    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:22.463548 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463520    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\") pod \"fe8b6db7-8a8f-474d-a706-a2613b836308\" (UID: \"fe8b6db7-8a8f-474d-a706-a2613b836308\") "
Nov 03 12:38:22.463745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463684    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:22.463910 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.463887    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:22.464131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.464112    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.464242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.464134    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.464242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.464148    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.464405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.464370    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:22.464485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.464465    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:22.466920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.466890    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "kube-api-access-vpdfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:22.467190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.467154    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:22.468703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.468678    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:22.468828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.468806    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token" pod "fe8b6db7-8a8f-474d-a706-a2613b836308" (UID: "fe8b6db7-8a8f-474d-a706-a2613b836308"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:22.565369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565328    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.565369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565371    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe8b6db7-8a8f-474d-a706-a2613b836308-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.565369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565403    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.565629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565418    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe8b6db7-8a8f-474d-a706-a2613b836308-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.565629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565434    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe8b6db7-8a8f-474d-a706-a2613b836308-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:22.565629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:22.565448    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpdfc\" (UniqueName: \"kubernetes.io/projected/fe8b6db7-8a8f-474d-a706-a2613b836308-kube-api-access-vpdfc\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:23.059562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.059530    5784 generic.go:356] "Generic (PLEG): container finished" podID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerID="6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e" exitCode=0
Nov 03 12:38:23.059746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.059581    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"
Nov 03 12:38:23.059746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.059616    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" event={"ID":"fe8b6db7-8a8f-474d-a706-a2613b836308","Type":"ContainerDied","Data":"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"}
Nov 03 12:38:23.059746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.059675    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf" event={"ID":"fe8b6db7-8a8f-474d-a706-a2613b836308","Type":"ContainerDied","Data":"c794e207d894449053955d72d34b82da6ee19096ec32c24a505b857405a56799"}
Nov 03 12:38:23.059746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.059702    5784 scope.go:122] "RemoveContainer" containerID="6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"
Nov 03 12:38:23.070686 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.070668    5784 scope.go:122] "RemoveContainer" containerID="6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"
Nov 03 12:38:23.071011 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:38:23.070991    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e\": container with ID starting with 6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e not found: ID does not exist" containerID="6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"
Nov 03 12:38:23.071069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.071020    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e"} err="failed to get container status \"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e\": rpc error: code = NotFound desc = could not find container \"6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e\": container with ID starting with 6abb76b0d09fea5a02ab1af2ef5dc434e7434b3f860300b90434aafd8230238e not found: ID does not exist"
Nov 03 12:38:23.260314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.260280    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:38:23.263366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.263335    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8d2c7c-openshift-default-669bc4dc75-vbmbf"]
Nov 03 12:38:23.456639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.456609    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:23.681709 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:23.681675    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" path="/var/lib/kubelet/pods/fe8b6db7-8a8f-474d-a706-a2613b836308/volumes"
Nov 03 12:38:26.472662 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:38:26.472633    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:38:27.833848 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:27.833807    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:29.955583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:29.952227    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:32.307607 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:32.307571    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:34.799571 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:34.799476    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:36.718049 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:36.718010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:41.088929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:41.088893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:44.124850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:44.124817    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:38:44.125334 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:44.125070    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy" containerID="cri-o://04b51835a7e8a1bff684182fe46077dc51f95a772e08a4e6c2c4fa3ab3fb2b5a" gracePeriod=30
Nov 03 12:38:45.118097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:45.118064    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:45.579883 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:45.579839    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:49.259605 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.259572    5784 generic.go:356] "Generic (PLEG): container finished" podID="2736721b-da3a-4fd0-a986-21e47e01765c" containerID="04b51835a7e8a1bff684182fe46077dc51f95a772e08a4e6c2c4fa3ab3fb2b5a" exitCode=0
Nov 03 12:38:49.260076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.259623    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" event={"ID":"2736721b-da3a-4fd0-a986-21e47e01765c","Type":"ContainerDied","Data":"04b51835a7e8a1bff684182fe46077dc51f95a772e08a4e6c2c4fa3ab3fb2b5a"}
Nov 03 12:38:49.389449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.389424    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:38:49.491750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491661    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.491920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491767    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.491920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491796    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.491920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491840    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.491920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491880    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.492134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.491923    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.492134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492043    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.492134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492054    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.492134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492096    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.492342 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492141    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\") pod \"2736721b-da3a-4fd0-a986-21e47e01765c\" (UID: \"2736721b-da3a-4fd0-a986-21e47e01765c\") "
Nov 03 12:38:49.492430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492338    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.492430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492358    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:38:49.492594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492568    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.492594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492582    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.492772 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492747    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.492906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492775    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2736721b-da3a-4fd0-a986-21e47e01765c-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.492906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492793    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.492906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492808    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.492906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.492822    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.494076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.494053    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:38:49.494181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.494091    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.494181 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.494095    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:38:49.494422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.494396    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m" pod "2736721b-da3a-4fd0-a986-21e47e01765c" (UID: "2736721b-da3a-4fd0-a986-21e47e01765c"). InnerVolumeSpecName "kube-api-access-6c65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:38:49.593475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.593435    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.593475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.593465    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c65m\" (UniqueName: \"kubernetes.io/projected/2736721b-da3a-4fd0-a986-21e47e01765c-kube-api-access-6c65m\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.593475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.593480    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2736721b-da3a-4fd0-a986-21e47e01765c-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.593727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.593492    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2736721b-da3a-4fd0-a986-21e47e01765c-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:38:49.912968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:49.912931    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:50.268003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:50.267913    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r" event={"ID":"2736721b-da3a-4fd0-a986-21e47e01765c","Type":"ContainerDied","Data":"be491c4c01e62c8dbaa028d3609b4ed02a39225b882bb9579386286400cf7de2"}
Nov 03 12:38:50.268003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:50.267936    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"
Nov 03 12:38:50.268003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:50.267977    5784 scope.go:122] "RemoveContainer" containerID="04b51835a7e8a1bff684182fe46077dc51f95a772e08a4e6c2c4fa3ab3fb2b5a"
Nov 03 12:38:50.430554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:50.430518    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:38:50.433640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:50.433612    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-96t6r"]
Nov 03 12:38:51.681720 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:51.681683    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" path="/var/lib/kubelet/pods/2736721b-da3a-4fd0-a986-21e47e01765c/volumes"
Nov 03 12:38:54.368144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:54.368113    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:56.154351 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:56.154307    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:56.309403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:56.309342    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:38:56.474424 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:38:56.474318    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:38:58.739614 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:38:58.739586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:01.202048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:01.202011    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:39:01.202533 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:01.202226    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy" containerID="cri-o://9f1e8678d1a955d143b5b07f4934b37bc7457a84d14355d2923a3a26a48ab949" gracePeriod=30
Nov 03 12:39:03.122205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:03.122160    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:06.403694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.403649    5784 generic.go:356] "Generic (PLEG): container finished" podID="18fdf350-b561-4322-a869-d8ed0613ba76" containerID="9f1e8678d1a955d143b5b07f4934b37bc7457a84d14355d2923a3a26a48ab949" exitCode=0
Nov 03 12:39:06.404164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.403810    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" event={"ID":"18fdf350-b561-4322-a869-d8ed0613ba76","Type":"ContainerDied","Data":"9f1e8678d1a955d143b5b07f4934b37bc7457a84d14355d2923a3a26a48ab949"}
Nov 03 12:39:06.478004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.477941    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:39:06.598445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598410    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598445    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598467    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598522    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598584    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598644    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598684    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598700    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598763    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598773    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598840    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:06.598925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.598898    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\") pod \"18fdf350-b561-4322-a869-d8ed0613ba76\" (UID: \"18fdf350-b561-4322-a869-d8ed0613ba76\") "
Nov 03 12:39:06.599241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599168    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:06.599241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599206    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:39:06.599485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599440    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18fdf350-b561-4322-a869-d8ed0613ba76-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.599485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599460    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.599485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599478    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.599736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599492    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.599736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.599505    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.600970 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.600937    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:39:06.601103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.601076    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "kube-api-access-6pw6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:39:06.601318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.601293    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:39:06.601428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.601312    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token" pod "18fdf350-b561-4322-a869-d8ed0613ba76" (UID: "18fdf350-b561-4322-a869-d8ed0613ba76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:39:06.700994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.700960    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pw6r\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-kube-api-access-6pw6r\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.700994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.700992    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18fdf350-b561-4322-a869-d8ed0613ba76-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.700994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.701003    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18fdf350-b561-4322-a869-d8ed0613ba76-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:06.701231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:06.701011    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18fdf350-b561-4322-a869-d8ed0613ba76-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:39:07.411432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.411404    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"
Nov 03 12:39:07.411872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.411398    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2" event={"ID":"18fdf350-b561-4322-a869-d8ed0613ba76","Type":"ContainerDied","Data":"41ca5b49da528eee9cc99d70375e3b95923f77138f0290831aeb77d34fd9ea7a"}
Nov 03 12:39:07.411872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.411534    5784 scope.go:122] "RemoveContainer" containerID="9f1e8678d1a955d143b5b07f4934b37bc7457a84d14355d2923a3a26a48ab949"
Nov 03 12:39:07.516667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.516625    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:07.577489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.577456    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:39:07.624919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.624885    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-tgpb2"]
Nov 03 12:39:07.681639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:07.681563    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" path="/var/lib/kubelet/pods/18fdf350-b561-4322-a869-d8ed0613ba76/volumes"
Nov 03 12:39:11.912631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:11.912589    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:16.272467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:16.272431    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:17.185259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:17.185223    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:20.713238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:20.713205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:25.207900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:25.207863    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:26.475179 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:39:26.475140    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:39:29.626101 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:29.626054    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:33.985395 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:33.985345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:36.487052 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:36.487018    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:37.939772 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:37.939735    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:38.496865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:38.496825    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:42.912593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:42.912507    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:47.358141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:47.358097    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:51.704634 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:51.704594    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:56.110641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:56.110605    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:56.465095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:39:56.465061    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:39:56.472767 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:39:56.472724    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:40:00.508651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:00.508620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:04.987041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:04.987000    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:09.368129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:09.368087    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:11.266941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:11.266906    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:13.775576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:13.775529    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:14.244133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:14.244090    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:14.705426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:14.705357    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:16.296570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:16.296535    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:18.185801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:18.185761    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:19.728783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:19.728751    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:22.605549 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:22.605507    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:26.474115 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:40:26.474066    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:40:27.029865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.029828    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:27.117451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.116363    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:40:27.117681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.117648    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery" containerID="cri-o://936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7" gracePeriod=30
Nov 03 12:40:27.186428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.186396    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:27.396005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.395976    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:40:27.474110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474067    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474104    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474144    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474176    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474209    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474263    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474346    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\") pod \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\" (UID: \"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4\") "
Nov 03 12:40:27.474725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474593    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:40:27.475053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.474967    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-ca-configmap\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.476819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.476778    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "kube-api-access-5czkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:40:27.476819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.476795    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:40:27.477053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.477029    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:27.477209 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.477186    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:27.477209 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.477201    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:40:27.478734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.478711    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token" pod "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" (UID: "4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:40:27.576502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576412    5784 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-local-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.576502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576454    5784 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-kubeconfig\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.576502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576469    5784 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-cacerts\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.576502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576482    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.576502 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576496    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-istio-csr-dns-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:27.576784 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:27.576512    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5czkc\" (UniqueName: \"kubernetes.io/projected/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4-kube-api-access-5czkc\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:40:28.002290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.002251    5784 generic.go:356] "Generic (PLEG): container finished" podID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerID="936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7" exitCode=0
Nov 03 12:40:28.002509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.002311    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"
Nov 03 12:40:28.002509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.002320    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" event={"ID":"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4","Type":"ContainerDied","Data":"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"}
Nov 03 12:40:28.002509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.002397    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx" event={"ID":"4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4","Type":"ContainerDied","Data":"6ae61030cd4ddf8c7b2ea2c685cc2f56bcdc466bd77ab5c506410fcc05fb0494"}
Nov 03 12:40:28.002509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.002431    5784 scope.go:122] "RemoveContainer" containerID="936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"
Nov 03 12:40:28.013705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.013684    5784 scope.go:122] "RemoveContainer" containerID="936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"
Nov 03 12:40:28.013999 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:40:28.013980    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7\": container with ID starting with 936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7 not found: ID does not exist" containerID="936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"
Nov 03 12:40:28.014063 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.014008    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7"} err="failed to get container status \"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7\": rpc error: code = NotFound desc = could not find container \"936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7\": container with ID starting with 936e0ad82e349592471769782883b744fd7b4d82e51dbf544b7caba8b40b59a7 not found: ID does not exist"
Nov 03 12:40:28.155244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.155207    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:40:28.200798 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:28.200765    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gdvvx"]
Nov 03 12:40:29.683611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:29.683569    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" path="/var/lib/kubelet/pods/4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4/volumes"
Nov 03 12:40:31.536102 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:31.536057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:35.919799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:35.919751    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:40.315677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:40.315634    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:44.714924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:44.714893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:49.099767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:49.099731    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:53.151702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:53.151650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:53.484752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:53.484661    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:53.642779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:53.642745    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:54.865023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:54.864977    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:55.123280 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:55.123192    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:56.443837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:56.443806    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:56.471827 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:40:56.471786    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:40:56.777996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:56.777913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:57.093213 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:57.093137    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:57.250229 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:57.250185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:58.236196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:58.236163    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:40:59.003947 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:40:59.003913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:01.948653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:01.948607    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:02.927353 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:02.927315    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:04.571158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:04.571126    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:07.316800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:07.316761    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:11.702538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:11.702452    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:15.530031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:15.529994    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:15.803437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:15.803316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:17.213203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:17.213168    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:17.727853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:17.727807    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:19.671481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:19.671447    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:41:19.672046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:19.671658    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy" containerID="cri-o://251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7" gracePeriod=30
Nov 03 12:41:19.717750 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:19.717721    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:41:19.717960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:19.717922    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy" containerID="cri-o://cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01" gracePeriod=30
Nov 03 12:41:21.626701 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:21.626668    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:21.772872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:21.772843    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:23.920851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:23.920815    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:24.088078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:24.088046    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:24.983235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:24.983205    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:41:25.013447 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.013426    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:41:25.061254 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061228    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.061407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061266    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.061407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061296    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061324    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.061407 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061358    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061416    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061453    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061490    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061531    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061560    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061576    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061591    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061618    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.061714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061658    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061723    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061764    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061797    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061841    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\") pod \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\" (UID: \"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061870    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\") pod \"be48a316-7d60-4ba9-81d5-cbd05632cd00\" (UID: \"be48a316-7d60-4ba9-81d5-cbd05632cd00\") "
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061899    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.062186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.061987    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.062585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062283    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.062585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062522    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.062585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062578    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.062752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062600    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.062752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062616    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.062752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062631    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.062752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.062709    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.063350 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.063005    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:25.063350 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.063319    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.064644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.063691    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.064644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.064279    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.064644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.064476    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:25.064960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.064916    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:25.065221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.065179    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:25.065483 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.065346    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "kube-api-access-mqjwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:25.065483 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.065399    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:25.065483 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.065417    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "kube-api-access-p58kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:25.065786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.065767    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert" pod "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" (UID: "89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:25.066723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.066695    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token" pod "be48a316-7d60-4ba9-81d5-cbd05632cd00" (UID: "be48a316-7d60-4ba9-81d5-cbd05632cd00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:25.164173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164135    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164172    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164188    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjwt\" (UniqueName: \"kubernetes.io/projected/be48a316-7d60-4ba9-81d5-cbd05632cd00-kube-api-access-mqjwt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164203    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164219    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be48a316-7d60-4ba9-81d5-cbd05632cd00-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164233    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164247    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164261    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164274    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164289    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be48a316-7d60-4ba9-81d5-cbd05632cd00-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164302    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164316    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p58kb\" (UniqueName: \"kubernetes.io/projected/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-kube-api-access-p58kb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164335    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.164439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.164348    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:25.381156 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.381125    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:25.421448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.418795    5784 generic.go:356] "Generic (PLEG): container finished" podID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerID="cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01" exitCode=0
Nov 03 12:41:25.421448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.419007    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" event={"ID":"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7","Type":"ContainerDied","Data":"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"}
Nov 03 12:41:25.421448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.419077    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96" event={"ID":"89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7","Type":"ContainerDied","Data":"c1edd01291a8e9193f243390c656f9685732b67d226d06a55c9b5523ef189e13"}
Nov 03 12:41:25.421448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.419113    5784 scope.go:122] "RemoveContainer" containerID="cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"
Nov 03 12:41:25.421448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.419317    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"
Nov 03 12:41:25.427543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.427505    5784 generic.go:356] "Generic (PLEG): container finished" podID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerID="251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7" exitCode=0
Nov 03 12:41:25.427684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.427559    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"
Nov 03 12:41:25.427684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.427588    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" event={"ID":"be48a316-7d60-4ba9-81d5-cbd05632cd00","Type":"ContainerDied","Data":"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"}
Nov 03 12:41:25.427684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.427637    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z" event={"ID":"be48a316-7d60-4ba9-81d5-cbd05632cd00","Type":"ContainerDied","Data":"f11e51b6c01ac498c649a8db726100a3d5fa29fa0f10d4934a9885096fd2f20d"}
Nov 03 12:41:25.438029 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.438000    5784 scope.go:122] "RemoveContainer" containerID="cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"
Nov 03 12:41:25.438307 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:41:25.438279    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01\": container with ID starting with cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01 not found: ID does not exist" containerID="cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"
Nov 03 12:41:25.438430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.438317    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01"} err="failed to get container status \"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01\": rpc error: code = NotFound desc = could not find container \"cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01\": container with ID starting with cdb566cbf41a503e2be35b88b468f9161a650b15dca0d290ae6d0ef9b28c0c01 not found: ID does not exist"
Nov 03 12:41:25.438430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.438341    5784 scope.go:122] "RemoveContainer" containerID="251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"
Nov 03 12:41:25.447597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.447580    5784 scope.go:122] "RemoveContainer" containerID="251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"
Nov 03 12:41:25.447860 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:41:25.447841    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7\": container with ID starting with 251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7 not found: ID does not exist" containerID="251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"
Nov 03 12:41:25.447912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.447868    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7"} err="failed to get container status \"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7\": rpc error: code = NotFound desc = could not find container \"251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7\": container with ID starting with 251367ae72e0137ddaf56b8345b00ae65107edc80399262fad092bb93a933be7 not found: ID does not exist"
Nov 03 12:41:25.557804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.557773    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:41:25.558080 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.558052    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" podUID="de5a1383-a832-4700-8d73-1f88e294d5a5" containerName="maas-api" containerID="cri-o://536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7" gracePeriod=30
Nov 03 12:41:25.576874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.576845    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:41:25.581267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.581245    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k6g4z"]
Nov 03 12:41:25.684674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.684646    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" path="/var/lib/kubelet/pods/be48a316-7d60-4ba9-81d5-cbd05632cd00/volumes"
Nov 03 12:41:25.685058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.685035    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:41:25.685175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.685076    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bb1b51-openshift-default-65dcdbc9b5-7wr96"]
Nov 03 12:41:25.685942 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.685916    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:25.852644 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.852610    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:41:25.993085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.993008    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\" (UniqueName: \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\") pod \"de5a1383-a832-4700-8d73-1f88e294d5a5\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") "
Nov 03 12:41:25.993085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.993056    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") pod \"de5a1383-a832-4700-8d73-1f88e294d5a5\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") "
Nov 03 12:41:25.993573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.993115    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\") pod \"de5a1383-a832-4700-8d73-1f88e294d5a5\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") "
Nov 03 12:41:25.993573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.993247    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\") pod \"de5a1383-a832-4700-8d73-1f88e294d5a5\" (UID: \"de5a1383-a832-4700-8d73-1f88e294d5a5\") "
Nov 03 12:41:25.993690 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.993607    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca" pod "de5a1383-a832-4700-8d73-1f88e294d5a5" (UID: "de5a1383-a832-4700-8d73-1f88e294d5a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:25.995323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.995299    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4" pod "de5a1383-a832-4700-8d73-1f88e294d5a5" (UID: "de5a1383-a832-4700-8d73-1f88e294d5a5"). InnerVolumeSpecName "kube-api-access-rc8s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:25.995671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.995643    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs" pod "de5a1383-a832-4700-8d73-1f88e294d5a5" (UID: "de5a1383-a832-4700-8d73-1f88e294d5a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:41:25.995671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:25.995655    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls" pod "de5a1383-a832-4700-8d73-1f88e294d5a5" (UID: "de5a1383-a832-4700-8d73-1f88e294d5a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:41:26.004587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.004563    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:26.094637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.094602    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:26.094637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.094637    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc8s4\" (UniqueName: \"kubernetes.io/projected/de5a1383-a832-4700-8d73-1f88e294d5a5-kube-api-access-rc8s4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:26.094877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.094654    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de5a1383-a832-4700-8d73-1f88e294d5a5-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:26.094877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.094669    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de5a1383-a832-4700-8d73-1f88e294d5a5-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:26.437343 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.437309    5784 generic.go:356] "Generic (PLEG): container finished" podID="de5a1383-a832-4700-8d73-1f88e294d5a5" containerID="536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7" exitCode=0
Nov 03 12:41:26.437566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.437361    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"
Nov 03 12:41:26.437566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.437411    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" event={"ID":"de5a1383-a832-4700-8d73-1f88e294d5a5","Type":"ContainerDied","Data":"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"}
Nov 03 12:41:26.437566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.437478    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr" event={"ID":"de5a1383-a832-4700-8d73-1f88e294d5a5","Type":"ContainerDied","Data":"914990b9143f888525aa3fb48d17c670eeee03b405194462c3c849b53c788ec8"}
Nov 03 12:41:26.437566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.437509    5784 scope.go:122] "RemoveContainer" containerID="536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"
Nov 03 12:41:26.488723 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:41:26.488675    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:41:26.488902 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.488748    5784 scope.go:122] "RemoveContainer" containerID="536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"
Nov 03 12:41:26.489132 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:41:26.489108    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7\": container with ID starting with 536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7 not found: ID does not exist" containerID="536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"
Nov 03 12:41:26.489198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.489144    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7"} err="failed to get container status \"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7\": rpc error: code = NotFound desc = could not find container \"536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7\": container with ID starting with 536b44442b6b4bf5ecaedf6899557a23cd88c0692b88e4b965a607f9ef14a9d7 not found: ID does not exist"
Nov 03 12:41:26.590162 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.590129    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:41:26.593108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.593080    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-561bd4-5c996f95d4-thwlr"]
Nov 03 12:41:26.809141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:26.809044    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:27.359666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:27.359620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:27.682296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:27.682257    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" path="/var/lib/kubelet/pods/89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7/volumes"
Nov 03 12:41:27.682785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:27.682769    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5a1383-a832-4700-8d73-1f88e294d5a5" path="/var/lib/kubelet/pods/de5a1383-a832-4700-8d73-1f88e294d5a5/volumes"
Nov 03 12:41:30.356221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:30.356185    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:41:30.356723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:30.356412    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" containerID="cri-o://e8ef39332a855587f478d307f1cfaf0d8a0e2b063b5918827cd1c4d568166878" gracePeriod=30
Nov 03 12:41:31.840511 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:31.840476    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:32.555664 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:32.555623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:33.464865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:33.464833    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:33.613105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:33.613055    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:34.128883 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.128854    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:34.232961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.232932    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:41:34.246778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.246754    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:41:34.280264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.280236    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:34.644998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.644959    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:41:34.645340 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:34.645021    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:41:35.512449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.512410    5784 generic.go:356] "Generic (PLEG): container finished" podID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerID="e8ef39332a855587f478d307f1cfaf0d8a0e2b063b5918827cd1c4d568166878" exitCode=0
Nov 03 12:41:35.512643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.512465    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" event={"ID":"038643c5-8fdf-4b50-bad8-4ee059945d89","Type":"ContainerDied","Data":"e8ef39332a855587f478d307f1cfaf0d8a0e2b063b5918827cd1c4d568166878"}
Nov 03 12:41:35.619745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.619721    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:41:35.629590 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.629566    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:35.805176 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805139    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805214    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805258    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805295    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805397    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805461    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805515    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805542    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.805679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.805636    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\") pod \"038643c5-8fdf-4b50-bad8-4ee059945d89\" (UID: \"038643c5-8fdf-4b50-bad8-4ee059945d89\") "
Nov 03 12:41:35.806111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806006    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:35.806169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806090    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.806228 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806180    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:35.806279 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806264    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:35.806327 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806288    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:35.806965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.806920    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:41:35.807952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.807913    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:41:35.808333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.808309    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:35.809526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.809503    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:41:35.812157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.812133    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6" pod "038643c5-8fdf-4b50-bad8-4ee059945d89" (UID: "038643c5-8fdf-4b50-bad8-4ee059945d89"). InnerVolumeSpecName "kube-api-access-rjlb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906785    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlb6\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-kube-api-access-rjlb6\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906813    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906822    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906831    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906839    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/038643c5-8fdf-4b50-bad8-4ee059945d89-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906847    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/038643c5-8fdf-4b50-bad8-4ee059945d89-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.906859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906855    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:35.907179 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:35.906865    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/038643c5-8fdf-4b50-bad8-4ee059945d89-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:41:36.121833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.121794    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:36.431597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.431562    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:36.519913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.519885    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"
Nov 03 12:41:36.520082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.519915    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5" event={"ID":"038643c5-8fdf-4b50-bad8-4ee059945d89","Type":"ContainerDied","Data":"1e02cc3361ed5f3df5987c88cd23468ea40180e75db38e1076946dba1bd4e59f"}
Nov 03 12:41:36.520082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.519973    5784 scope.go:122] "RemoveContainer" containerID="e8ef39332a855587f478d307f1cfaf0d8a0e2b063b5918827cd1c4d568166878"
Nov 03 12:41:36.672877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.672836    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:41:36.680632 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.680603    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-wskf5"]
Nov 03 12:41:36.751255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:36.751154    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:37.344168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:37.344134    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:37.610391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:37.610285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:37.682306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:37.682265    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" path="/var/lib/kubelet/pods/038643c5-8fdf-4b50-bad8-4ee059945d89/volumes"
Nov 03 12:41:38.228879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:38.228839    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:41.216663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:41.216634    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:41.603684 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:41.603602    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:41.774363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:41.774334    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:42.019430 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:42.019388    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:44.640429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:44.640365    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:44.801937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:44.801904    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:46.026691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:46.026658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:50.562969 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:50.562930    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:55.015581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:55.015542    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:56.471837 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:41:56.471790    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:41:57.093433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:57.093397    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:41:59.430511 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:41:59.430483    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:01.163081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:01.163044    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:01.310418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:01.310371    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:03.835064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:03.835028    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:04.981124 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:04.981088    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:08.206097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:08.206057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:12.713576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:12.713542    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:17.088074 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:17.088038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:21.475735 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:21.475695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:25.846255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:25.846222    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:26.472720 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:42:26.472678    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:42:29.599807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.599770    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:42:29.600579 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.599959    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" podUID="52af2b97-eb5e-4558-a649-5571fa899d02" containerName="maas-api" containerID="cri-o://6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b" gracePeriod=30
Nov 03 12:42:29.739133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.739097    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:29.866946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.866885    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:42:29.879565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.879539    5784 generic.go:356] "Generic (PLEG): container finished" podID="52af2b97-eb5e-4558-a649-5571fa899d02" containerID="6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b" exitCode=0
Nov 03 12:42:29.879697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.879586    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"
Nov 03 12:42:29.879697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.879624    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" event={"ID":"52af2b97-eb5e-4558-a649-5571fa899d02","Type":"ContainerDied","Data":"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"}
Nov 03 12:42:29.879819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.879707    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm" event={"ID":"52af2b97-eb5e-4558-a649-5571fa899d02","Type":"ContainerDied","Data":"6c78ba00bcaa7985310be68354d0018fba697f41cd8232c81a053207170fcde6"}
Nov 03 12:42:29.879819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.879733    5784 scope.go:122] "RemoveContainer" containerID="6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"
Nov 03 12:42:29.891573 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.891552    5784 scope.go:122] "RemoveContainer" containerID="6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"
Nov 03 12:42:29.891894 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:42:29.891873    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b\": container with ID starting with 6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b not found: ID does not exist" containerID="6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"
Nov 03 12:42:29.891982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:29.891903    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b"} err="failed to get container status \"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b\": rpc error: code = NotFound desc = could not find container \"6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b\": container with ID starting with 6f09d3fb1af5f3ea37a95bf97e3df7ea1f2f1c39da7666e876b369a9291cf95b not found: ID does not exist"
Nov 03 12:42:30.021299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.021266    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") pod \"52af2b97-eb5e-4558-a649-5571fa899d02\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") "
Nov 03 12:42:30.021537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.021441    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\" (UniqueName: \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\") pod \"52af2b97-eb5e-4558-a649-5571fa899d02\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") "
Nov 03 12:42:30.021628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.021549    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\") pod \"52af2b97-eb5e-4558-a649-5571fa899d02\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") "
Nov 03 12:42:30.021628 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.021578    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\") pod \"52af2b97-eb5e-4558-a649-5571fa899d02\" (UID: \"52af2b97-eb5e-4558-a649-5571fa899d02\") "
Nov 03 12:42:30.022256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.022224    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca" pod "52af2b97-eb5e-4558-a649-5571fa899d02" (UID: "52af2b97-eb5e-4558-a649-5571fa899d02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:42:30.023744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.023715    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs" pod "52af2b97-eb5e-4558-a649-5571fa899d02" (UID: "52af2b97-eb5e-4558-a649-5571fa899d02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:42:30.023863 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.023794    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md" pod "52af2b97-eb5e-4558-a649-5571fa899d02" (UID: "52af2b97-eb5e-4558-a649-5571fa899d02"). InnerVolumeSpecName "kube-api-access-r82md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:42:30.024096 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.024068    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls" pod "52af2b97-eb5e-4558-a649-5571fa899d02" (UID: "52af2b97-eb5e-4558-a649-5571fa899d02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:42:30.123358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.123279    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:30.123358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.123312    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82md\" (UniqueName: \"kubernetes.io/projected/52af2b97-eb5e-4558-a649-5571fa899d02-kube-api-access-r82md\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:30.123358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.123323    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52af2b97-eb5e-4558-a649-5571fa899d02-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:30.123358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.123331    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52af2b97-eb5e-4558-a649-5571fa899d02-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:42:30.292132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.292097    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:42:30.295930 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:30.295898    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-3176d9-54f6884fdf-lqswm"]
Nov 03 12:42:31.315853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:31.315820    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:31.682535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:31.682494    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52af2b97-eb5e-4558-a649-5571fa899d02" path="/var/lib/kubelet/pods/52af2b97-eb5e-4558-a649-5571fa899d02/volumes"
Nov 03 12:42:35.303620 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.303587    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:35.437044 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437009    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:42:35.437759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437732    5784 cpu_manager.go:414] "containerMap: removing container" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy"
Nov 03 12:42:35.437759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437753    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437774    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437782    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437790    5784 cpu_manager.go:414] "containerMap: removing container" podUID="52af2b97-eb5e-4558-a649-5571fa899d02" containerName="maas-api"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437798    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52af2b97-eb5e-4558-a649-5571fa899d02" containerName="maas-api"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437820    5784 cpu_manager.go:414] "containerMap: removing container" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437825    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437837    5784 cpu_manager.go:414] "containerMap: removing container" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437841    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437855    5784 cpu_manager.go:414] "containerMap: removing container" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437862    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437873    5784 cpu_manager.go:414] "containerMap: removing container" podUID="de5a1383-a832-4700-8d73-1f88e294d5a5" containerName="maas-api"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437879    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5a1383-a832-4700-8d73-1f88e294d5a5" containerName="maas-api"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437898    5784 cpu_manager.go:414] "containerMap: removing container" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437905    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy"
Nov 03 12:42:35.437916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437920    5784 cpu_manager.go:414] "containerMap: removing container" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.437927    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438042    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f5c72c-5d6b-4a81-b0d9-edde64c4b3d7" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438056    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5a1383-a832-4700-8d73-1f88e294d5a5" containerName="maas-api"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438064    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="18fdf350-b561-4322-a869-d8ed0613ba76" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438071    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="be48a316-7d60-4ba9-81d5-cbd05632cd00" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438078    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="038643c5-8fdf-4b50-bad8-4ee059945d89" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438085    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2736721b-da3a-4fd0-a986-21e47e01765c" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438091    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="52af2b97-eb5e-4558-a649-5571fa899d02" containerName="maas-api"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438100    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe8b6db7-8a8f-474d-a706-a2613b836308" containerName="istio-proxy"
Nov 03 12:42:35.438530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.438106    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc87c1a-fdb0-4cd2-9184-d4b8724d57d4" containerName="discovery"
Nov 03 12:42:35.445735 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.445707    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.461471 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.461443    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:35.494780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.494741    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:42:35.536914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.536879    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-52ab15-serving-cert\""
Nov 03 12:42:35.580303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.580205    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.580303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.580285    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.580575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.580407    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.580575 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.580439    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h47wf\" (UniqueName: \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.638347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.638316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:35.681334 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.681303    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.681655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.681621    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.681801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.681742    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.681801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.681776    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h47wf\" (UniqueName: \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.682048 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:42:35.682020    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-serving-cert: secret "maas-api-e2e-explicit-52ab15-serving-cert" not found
Nov 03 12:42:35.682131 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:42:35.682091    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls podName:c9052b6b-ccb4-47d6-a721-b646fa715791 nodeName:}" failed. No retries permitted until 2026-11-03 12:42:36.182071095 +0000 UTC m=+3062.624273398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls") pod "maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" (UID: "c9052b6b-ccb4-47d6-a721-b646fa715791") : secret "maas-api-e2e-explicit-52ab15-serving-cert" not found
Nov 03 12:42:35.682131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.682113    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.684814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.684791    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.735646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.735616    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h47wf\" (UniqueName: \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:35.805247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.805215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:35.997292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:35.997259    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:36.186258 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.186224    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:36.188988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.188953    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") pod \"maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:36.361987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.361909    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:36.730806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.729332    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:42:36.731092 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:42:36.731053    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9052b6b_ccb4_47d6_a721_b646fa715791.slice/crio-bb25279b8ea248f2815d3f10bd285eedea9428fe0e0c12208646c7d63d3f8b88 WatchSource:0}: Error finding container bb25279b8ea248f2815d3f10bd285eedea9428fe0e0c12208646c7d63d3f8b88: Status 404 returned error can't find the container with id bb25279b8ea248f2815d3f10bd285eedea9428fe0e0c12208646c7d63d3f8b88
Nov 03 12:42:36.732658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.732637    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:42:36.900095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.900036    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:36.927869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:36.927826    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" event={"ID":"c9052b6b-ccb4-47d6-a721-b646fa715791","Type":"ContainerStarted","Data":"bb25279b8ea248f2815d3f10bd285eedea9428fe0e0c12208646c7d63d3f8b88"}
Nov 03 12:42:37.422757 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:37.422717    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:37.938284 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:37.938246    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" event={"ID":"c9052b6b-ccb4-47d6-a721-b646fa715791","Type":"ContainerStarted","Data":"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"}
Nov 03 12:42:37.938486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:37.938354    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:38.080833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:38.080736    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" podStartSLOduration=1.999802979 podStartE2EDuration="3.080721248s" totalImagesPullingTime="1.080918269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:38.078839178 +0000 UTC m=+3064.521041483" watchObservedRunningTime="2026-11-03 12:42:38.080721248 +0000 UTC m=+3064.522923560"
Nov 03 12:42:38.228426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:38.228366    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:38.798030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:38.797984    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:38.973185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:38.973155    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:40.414085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:40.414054    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:43.065802 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:43.065727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:43.224802 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:43.224772    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:43.389842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:43.389814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:43.951730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:43.951701    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:42:45.240994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:45.240956    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:46.277204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.277174    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:46.445344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.445305    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:46.661152 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.661115    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 12:42:46.665950 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.665924    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.702232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.702188    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb69r\" (UniqueName: \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.702728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.702700    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.716197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.716140    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 12:42:46.758431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.758386    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a4489c-dockercfg-mgq67\""
Nov 03 12:42:46.758603 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.758462    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a4489c\""
Nov 03 12:42:46.793567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.793531    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:46.804137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.804104    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.804285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.804181    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb69r\" (UniqueName: \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.804825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.804807    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.825949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.825908    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 12:42:46.830988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.830954    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:46.870451 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.868341    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb69r\" (UniqueName: \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\") pod \"payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:46.887803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.887770    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 12:42:46.905871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.905835    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:46.905871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.905876    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rms97\" (UniqueName: \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:46.979785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:46.979698    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:47.012046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.012001    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:47.012225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.012068    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rms97\" (UniqueName: \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:47.013314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.013277    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:47.070782 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.070745    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rms97\" (UniqueName: \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\") pod \"payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:47.179953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.179911    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:47.395403 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.395357    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 12:42:47.397492 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:42:47.397458    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddced03_6163_4a21_9c27_7b5aa2784db1.slice/crio-d391b66d643fd02f9c9b2ba5c6761d4da25201a38432a81b2c0542c803e4a942 WatchSource:0}: Error finding container d391b66d643fd02f9c9b2ba5c6761d4da25201a38432a81b2c0542c803e4a942: Status 404 returned error can't find the container with id d391b66d643fd02f9c9b2ba5c6761d4da25201a38432a81b2c0542c803e4a942
Nov 03 12:42:47.552048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:47.551991    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 12:42:47.553282 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:42:47.553252    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0b7e1_c4c1_45db_abd6_669fc5e83a44.slice/crio-05d0159729893939453243dc0c82d0e1e5d1a88d8605a1d09822ea4f52476142 WatchSource:0}: Error finding container 05d0159729893939453243dc0c82d0e1e5d1a88d8605a1d09822ea4f52476142: Status 404 returned error can't find the container with id 05d0159729893939453243dc0c82d0e1e5d1a88d8605a1d09822ea4f52476142
Nov 03 12:42:48.011677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.011637    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" event={"ID":"89e0b7e1-c4c1-45db-abd6-669fc5e83a44","Type":"ContainerStarted","Data":"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"}
Nov 03 12:42:48.011865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.011700    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" event={"ID":"89e0b7e1-c4c1-45db-abd6-669fc5e83a44","Type":"ContainerStarted","Data":"05d0159729893939453243dc0c82d0e1e5d1a88d8605a1d09822ea4f52476142"}
Nov 03 12:42:48.011973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.011859    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:42:48.013432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.013405    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" event={"ID":"6ddced03-6163-4a21-9c27-7b5aa2784db1","Type":"ContainerStarted","Data":"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"}
Nov 03 12:42:48.013555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.013445    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" event={"ID":"6ddced03-6163-4a21-9c27-7b5aa2784db1","Type":"ContainerStarted","Data":"d391b66d643fd02f9c9b2ba5c6761d4da25201a38432a81b2c0542c803e4a942"}
Nov 03 12:42:48.013555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.013477    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:48.117425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.117341    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" podStartSLOduration=2.11732216 podStartE2EDuration="2.11732216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:48.115522781 +0000 UTC m=+3074.557725087" watchObservedRunningTime="2026-11-03 12:42:48.11732216 +0000 UTC m=+3074.559524466"
Nov 03 12:42:48.222003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:48.221946    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" podStartSLOduration=2.221929584 podStartE2EDuration="2.221929584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:42:48.219168347 +0000 UTC m=+3074.661370664" watchObservedRunningTime="2026-11-03 12:42:48.221929584 +0000 UTC m=+3074.664131890"
Nov 03 12:42:49.923849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:49.923808    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:54.301422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:54.301366    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:56.474217 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:42:56.474176    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:42:57.407076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:57.407040    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:58.680939 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:58.680900    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:42:59.024800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:59.024698    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 12:42:59.025109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:42:59.025088    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 12:43:03.110194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:03.110142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:06.188623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.188564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:06.406520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.406486    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-98e919"]
Nov 03 12:43:06.410466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.410438    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:06.433356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.433330    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9462\" (UniqueName: \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\") pod \"test-curl-7590-98e919\" (UID: \"8ae15531-b1be-4ee1-8976-55e8fb025a20\") " pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:06.460978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.460912    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-98e919"]
Nov 03 12:43:06.503532 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.503503    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 12:43:06.535272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.535238    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9462\" (UniqueName: \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\") pod \"test-curl-7590-98e919\" (UID: \"8ae15531-b1be-4ee1-8976-55e8fb025a20\") " pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:06.588509 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.588471    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9462\" (UniqueName: \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\") pod \"test-curl-7590-98e919\" (UID: \"8ae15531-b1be-4ee1-8976-55e8fb025a20\") " pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:06.730371 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:06.730278    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:07.103429 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:07.103398    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-98e919"]
Nov 03 12:43:07.105101 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:07.105069    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ae15531_b1be_4ee1_8976_55e8fb025a20.slice/crio-a6c1a1479e6359938bc98364e0465d1f9332e672c437ed097a10e14e96459d53 WatchSource:0}: Error finding container a6c1a1479e6359938bc98364e0465d1f9332e672c437ed097a10e14e96459d53: Status 404 returned error can't find the container with id a6c1a1479e6359938bc98364e0465d1f9332e672c437ed097a10e14e96459d53
Nov 03 12:43:07.145445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:07.145407    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-98e919" event={"ID":"8ae15531-b1be-4ee1-8976-55e8fb025a20","Type":"ContainerStarted","Data":"a6c1a1479e6359938bc98364e0465d1f9332e672c437ed097a10e14e96459d53"}
Nov 03 12:43:07.545197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:07.545161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:08.156258 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:08.156222    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-98e919" event={"ID":"8ae15531-b1be-4ee1-8976-55e8fb025a20","Type":"ContainerStarted","Data":"e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78"}
Nov 03 12:43:08.264456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:08.264371    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7590-98e919" podStartSLOduration=2.26435313 podStartE2EDuration="2.26435313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:08.262489964 +0000 UTC m=+3094.704692289" watchObservedRunningTime="2026-11-03 12:43:08.26435313 +0000 UTC m=+3094.706555436"
Nov 03 12:43:09.666293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:09.666221    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-98e919"]
Nov 03 12:43:09.669823 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:09.669763    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-98e919"]
Nov 03 12:43:10.038188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.038112    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-95ec63"]
Nov 03 12:43:10.038646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.038632    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" containerName="test-curl-7590-98e919"
Nov 03 12:43:10.038702 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.038647    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" containerName="test-curl-7590-98e919"
Nov 03 12:43:10.038763 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.038754    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" containerName="test-curl-7590-98e919"
Nov 03 12:43:10.042432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.042412    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:10.083065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.083036    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6hc\" (UniqueName: \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\") pod \"test-curl-7590-95ec63\" (UID: \"5210fe83-3b58-4d12-a102-a366d85000ed\") " pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:10.090966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.090941    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-95ec63"]
Nov 03 12:43:10.167859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.167823    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7590-98e919" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" containerName="test-curl-7590-98e919" containerID="cri-o://e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78" gracePeriod=2
Nov 03 12:43:10.184712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.184680    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6hc\" (UniqueName: \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\") pod \"test-curl-7590-95ec63\" (UID: \"5210fe83-3b58-4d12-a102-a366d85000ed\") " pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:10.237043 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.237015    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6hc\" (UniqueName: \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\") pod \"test-curl-7590-95ec63\" (UID: \"5210fe83-3b58-4d12-a102-a366d85000ed\") " pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:10.354733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.354658    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:10.724100 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:10.724073    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-95ec63"]
Nov 03 12:43:10.725655 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:10.725628    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5210fe83_3b58_4d12_a102_a366d85000ed.slice/crio-e5732f37c57bd7c2a153f6ba00320d6d63611f08006b28cac56d12e2130f5c17 WatchSource:0}: Error finding container e5732f37c57bd7c2a153f6ba00320d6d63611f08006b28cac56d12e2130f5c17: Status 404 returned error can't find the container with id e5732f37c57bd7c2a153f6ba00320d6d63611f08006b28cac56d12e2130f5c17
Nov 03 12:43:11.175307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:11.175267    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-95ec63" event={"ID":"5210fe83-3b58-4d12-a102-a366d85000ed","Type":"ContainerStarted","Data":"c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212"}
Nov 03 12:43:11.175307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:11.175307    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-95ec63" event={"ID":"5210fe83-3b58-4d12-a102-a366d85000ed","Type":"ContainerStarted","Data":"e5732f37c57bd7c2a153f6ba00320d6d63611f08006b28cac56d12e2130f5c17"}
Nov 03 12:43:11.279178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:11.279112    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7590-95ec63" podStartSLOduration=1.279092465 podStartE2EDuration="1.279092465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:11.27803973 +0000 UTC m=+3097.720242056" watchObservedRunningTime="2026-11-03 12:43:11.279092465 +0000 UTC m=+3097.721294771"
Nov 03 12:43:12.002968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.002933    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:12.402112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.402086    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:12.513068 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.513034    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\" (UniqueName: \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\") pod \"8ae15531-b1be-4ee1-8976-55e8fb025a20\" (UID: \"8ae15531-b1be-4ee1-8976-55e8fb025a20\") "
Nov 03 12:43:12.515058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.515030    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462" pod "8ae15531-b1be-4ee1-8976-55e8fb025a20" (UID: "8ae15531-b1be-4ee1-8976-55e8fb025a20"). InnerVolumeSpecName "kube-api-access-w9462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:12.596348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.596236    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-95ec63"]
Nov 03 12:43:12.600270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.600239    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-95ec63"]
Nov 03 12:43:12.614652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.614613    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9462\" (UniqueName: \"kubernetes.io/projected/8ae15531-b1be-4ee1-8976-55e8fb025a20-kube-api-access-w9462\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:12.753617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.753586    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-9e5891"]
Nov 03 12:43:12.754267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.754246    5784 cpu_manager.go:414] "containerMap: removing container" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" containerName="test-curl-7590-95ec63"
Nov 03 12:43:12.754267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.754265    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" containerName="test-curl-7590-95ec63"
Nov 03 12:43:12.754436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.754343    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" containerName="test-curl-7590-95ec63"
Nov 03 12:43:12.757639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.757623    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:12.807811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.807781    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-9e5891"]
Nov 03 12:43:12.817141 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.817107    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmn4\" (UniqueName: \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\") pod \"test-curl-7590-9e5891\" (UID: \"b207d93d-b8b0-4d44-9311-9aae29ae8f44\") " pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:12.917864 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.917823    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmn4\" (UniqueName: \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\") pod \"test-curl-7590-9e5891\" (UID: \"b207d93d-b8b0-4d44-9311-9aae29ae8f44\") " pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:12.970711 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:12.970675    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmn4\" (UniqueName: \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\") pod \"test-curl-7590-9e5891\" (UID: \"b207d93d-b8b0-4d44-9311-9aae29ae8f44\") " pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:13.069779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.069748    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:13.190751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.190664    5784 generic.go:356] "Generic (PLEG): container finished" podID="8ae15531-b1be-4ee1-8976-55e8fb025a20" containerID="e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78" exitCode=137
Nov 03 12:43:13.190751 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.190716    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:13.190981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.190758    5784 scope.go:122] "RemoveContainer" containerID="e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78"
Nov 03 12:43:13.190981 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.190950    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7590-95ec63" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" containerName="test-curl-7590-95ec63" containerID="cri-o://c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212" gracePeriod=2
Nov 03 12:43:13.200348 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.200329    5784 scope.go:122] "RemoveContainer" containerID="e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78"
Nov 03 12:43:13.200653 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.200636    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78\": container with ID starting with e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78 not found: ID does not exist" containerID="e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78"
Nov 03 12:43:13.200699 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.200661    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78"} err="failed to get container status \"e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78\": rpc error: code = NotFound desc = could not find container \"e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78\": container with ID starting with e937fa3d259e033aead8c602b2f47ad570d6925d6e99ec72803329d185feec78 not found: ID does not exist"
Nov 03 12:43:13.238083 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.238050    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-95ec63\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:13.285056 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.285019    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-98e919\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:13.332018 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.331979    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-95ec63\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:13.434670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.434645    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-9e5891"]
Nov 03 12:43:13.436293 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:13.436263    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb207d93d_b8b0_4d44_9311_9aae29ae8f44.slice/crio-942b22827cee50e38ac805fd49daff99f2caa8e7c89cbcf352d876a08ab9b44c WatchSource:0}: Error finding container 942b22827cee50e38ac805fd49daff99f2caa8e7c89cbcf352d876a08ab9b44c: Status 404 returned error can't find the container with id 942b22827cee50e38ac805fd49daff99f2caa8e7c89cbcf352d876a08ab9b44c
Nov 03 12:43:13.683290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:13.683252    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" path="/var/lib/kubelet/pods/8ae15531-b1be-4ee1-8976-55e8fb025a20/volumes"
Nov 03 12:43:13.726843 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.726745    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-98e919\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8ae15531-b1be-4ee1-8976-55e8fb025a20" pod="opendatahub/test-curl-7590-98e919"
Nov 03 12:43:13.773687 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:13.773645    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-95ec63\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:14.200467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:14.200425    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-9e5891" event={"ID":"b207d93d-b8b0-4d44-9311-9aae29ae8f44","Type":"ContainerStarted","Data":"3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a"}
Nov 03 12:43:14.200995 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:14.200475    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-9e5891" event={"ID":"b207d93d-b8b0-4d44-9311-9aae29ae8f44","Type":"ContainerStarted","Data":"942b22827cee50e38ac805fd49daff99f2caa8e7c89cbcf352d876a08ab9b44c"}
Nov 03 12:43:14.304832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:14.304776    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7590-9e5891" podStartSLOduration=2.304761785 podStartE2EDuration="2.304761785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:14.303302603 +0000 UTC m=+3100.745504925" watchObservedRunningTime="2026-11-03 12:43:14.304761785 +0000 UTC m=+3100.746964090"
Nov 03 12:43:15.474122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.474097    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:15.545659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.545630    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\" (UniqueName: \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\") pod \"5210fe83-3b58-4d12-a102-a366d85000ed\" (UID: \"5210fe83-3b58-4d12-a102-a366d85000ed\") "
Nov 03 12:43:15.547718 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.547688    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc" pod "5210fe83-3b58-4d12-a102-a366d85000ed" (UID: "5210fe83-3b58-4d12-a102-a366d85000ed"). InnerVolumeSpecName "kube-api-access-6n6hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:15.647556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.647523    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6hc\" (UniqueName: \"kubernetes.io/projected/5210fe83-3b58-4d12-a102-a366d85000ed-kube-api-access-6n6hc\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:15.681319 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.681286    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5210fe83-3b58-4d12-a102-a366d85000ed" path="/var/lib/kubelet/pods/5210fe83-3b58-4d12-a102-a366d85000ed/volumes"
Nov 03 12:43:15.730259 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.730179    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-9e5891"]
Nov 03 12:43:15.733440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.733415    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-9e5891"]
Nov 03 12:43:15.884960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.882909    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-b3c89f"]
Nov 03 12:43:15.884960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.884218    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" containerName="test-curl-7590-9e5891"
Nov 03 12:43:15.884960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.884246    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" containerName="test-curl-7590-9e5891"
Nov 03 12:43:15.884960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.884582    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" containerName="test-curl-7590-9e5891"
Nov 03 12:43:15.891469 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.891437    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:15.934637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.934609    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-b3c89f"]
Nov 03 12:43:15.950011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:15.949975    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7s2\" (UniqueName: \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\") pod \"test-curl-7590-b3c89f\" (UID: \"45608022-d8cf-452e-b602-17077ae8bee9\") " pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:16.051364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.051270    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7s2\" (UniqueName: \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\") pod \"test-curl-7590-b3c89f\" (UID: \"45608022-d8cf-452e-b602-17077ae8bee9\") " pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:16.103930 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.103900    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7s2\" (UniqueName: \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\") pod \"test-curl-7590-b3c89f\" (UID: \"45608022-d8cf-452e-b602-17077ae8bee9\") " pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:16.203529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.203492    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:16.215766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.215731    5784 generic.go:356] "Generic (PLEG): container finished" podID="5210fe83-3b58-4d12-a102-a366d85000ed" containerID="c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212" exitCode=137
Nov 03 12:43:16.215903 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.215796    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-95ec63"
Nov 03 12:43:16.216126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.216110    5784 scope.go:122] "RemoveContainer" containerID="c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212"
Nov 03 12:43:16.216500 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.216171    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7590-9e5891" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" containerName="test-curl-7590-9e5891" containerID="cri-o://3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a" gracePeriod=2
Nov 03 12:43:16.252082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.252054    5784 scope.go:122] "RemoveContainer" containerID="c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212"
Nov 03 12:43:16.252534 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:16.252505    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212\": container with ID starting with c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212 not found: ID does not exist" containerID="c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212"
Nov 03 12:43:16.252652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.252548    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212"} err="failed to get container status \"c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212\": rpc error: code = NotFound desc = could not find container \"c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212\": container with ID starting with c58c3f0589af14b566ccd5de093b8009e51f005196297738061234bdd4965212 not found: ID does not exist"
Nov 03 12:43:16.264018 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:16.263919    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-9e5891\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:16.311063 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:16.310984    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-9e5891\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:16.370372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.370336    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:16.572687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:16.572651    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-b3c89f"]
Nov 03 12:43:16.574087 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:16.574056    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45608022_d8cf_452e_b602_17077ae8bee9.slice/crio-e8fa98fa3ee2306695bc71fbdf63ad23ac2e230d789a97b9bc6e842bd185edf2 WatchSource:0}: Error finding container e8fa98fa3ee2306695bc71fbdf63ad23ac2e230d789a97b9bc6e842bd185edf2: Status 404 returned error can't find the container with id e8fa98fa3ee2306695bc71fbdf63ad23ac2e230d789a97b9bc6e842bd185edf2
Nov 03 12:43:17.226482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:17.226437    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-b3c89f" event={"ID":"45608022-d8cf-452e-b602-17077ae8bee9","Type":"ContainerStarted","Data":"cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae"}
Nov 03 12:43:17.226482 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:17.226490    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-b3c89f" event={"ID":"45608022-d8cf-452e-b602-17077ae8bee9","Type":"ContainerStarted","Data":"e8fa98fa3ee2306695bc71fbdf63ad23ac2e230d789a97b9bc6e842bd185edf2"}
Nov 03 12:43:17.273726 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:17.273689    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-9e5891\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:17.380788 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:17.380736    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7590-b3c89f" podStartSLOduration=2.380721162 podStartE2EDuration="2.380721162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:17.377415927 +0000 UTC m=+3103.819618232" watchObservedRunningTime="2026-11-03 12:43:17.380721162 +0000 UTC m=+3103.822923461"
Nov 03 12:43:18.086221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.086187    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:43:18.086929 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.086890    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" containerID="cri-o://684288b7f72de52788d8eaf7e39cdcd9b032ae7dc1331ee3389179dbd2c5aaaa" gracePeriod=30
Nov 03 12:43:18.452821 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.452791    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:18.577855 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.577824    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\" (UniqueName: \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\") pod \"b207d93d-b8b0-4d44-9311-9aae29ae8f44\" (UID: \"b207d93d-b8b0-4d44-9311-9aae29ae8f44\") "
Nov 03 12:43:18.579827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.579799    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4" pod "b207d93d-b8b0-4d44-9311-9aae29ae8f44" (UID: "b207d93d-b8b0-4d44-9311-9aae29ae8f44"). InnerVolumeSpecName "kube-api-access-jsmn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:18.679097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.679068    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsmn4\" (UniqueName: \"kubernetes.io/projected/b207d93d-b8b0-4d44-9311-9aae29ae8f44-kube-api-access-jsmn4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:18.713285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.713251    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-b3c89f"]
Nov 03 12:43:18.717524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.717472    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-b3c89f"]
Nov 03 12:43:18.864904 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.864864    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7590-29aa51"]
Nov 03 12:43:18.865450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.865431    5784 cpu_manager.go:414] "containerMap: removing container" podUID="45608022-d8cf-452e-b602-17077ae8bee9" containerName="test-curl-7590-b3c89f"
Nov 03 12:43:18.865555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.865451    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45608022-d8cf-452e-b602-17077ae8bee9" containerName="test-curl-7590-b3c89f"
Nov 03 12:43:18.865555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.865552    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="45608022-d8cf-452e-b602-17077ae8bee9" containerName="test-curl-7590-b3c89f"
Nov 03 12:43:18.868900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.868885    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:18.917413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.916555    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-29aa51"]
Nov 03 12:43:18.982761 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:18.982671    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nlh\" (UniqueName: \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\") pod \"test-curl-7590-29aa51\" (UID: \"e697b731-d027-4803-b0b8-57f525d0e2c1\") " pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:19.083331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.083294    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nlh\" (UniqueName: \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\") pod \"test-curl-7590-29aa51\" (UID: \"e697b731-d027-4803-b0b8-57f525d0e2c1\") " pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:19.135973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.135937    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nlh\" (UniqueName: \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\") pod \"test-curl-7590-29aa51\" (UID: \"e697b731-d027-4803-b0b8-57f525d0e2c1\") " pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:19.180486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.180454    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:19.268543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.268495    5784 generic.go:356] "Generic (PLEG): container finished" podID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" containerID="3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a" exitCode=137
Nov 03 12:43:19.268853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.268806    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7590-b3c89f" podUID="45608022-d8cf-452e-b602-17077ae8bee9" containerName="test-curl-7590-b3c89f" containerID="cri-o://cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae" gracePeriod=2
Nov 03 12:43:19.269002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.268984    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:19.269171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.269155    5784 scope.go:122] "RemoveContainer" containerID="3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a"
Nov 03 12:43:19.284977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.284954    5784 scope.go:122] "RemoveContainer" containerID="3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a"
Nov 03 12:43:19.285357 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:19.285335    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a\": container with ID starting with 3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a not found: ID does not exist" containerID="3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a"
Nov 03 12:43:19.285481 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.285365    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a"} err="failed to get container status \"3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a\": rpc error: code = NotFound desc = could not find container \"3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a\": container with ID starting with 3e8a61a76f06551e8216a17b7468f448dfc218006d9fdeb7d30665b30f28de4a not found: ID does not exist"
Nov 03 12:43:19.316179 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:19.316147    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-b3c89f\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="45608022-d8cf-452e-b602-17077ae8bee9" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:19.363121 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:19.363080    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-9e5891\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" pod="opendatahub/test-curl-7590-9e5891"
Nov 03 12:43:19.410039 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:19.410000    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-b3c89f\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="45608022-d8cf-452e-b602-17077ae8bee9" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:19.551704 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:19.551669    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode697b731_d027_4803_b0b8_57f525d0e2c1.slice/crio-5c689ca017aeeaaf3423c772c60cabd0259bbbcb53e7373276c97665247cfbc2 WatchSource:0}: Error finding container 5c689ca017aeeaaf3423c772c60cabd0259bbbcb53e7373276c97665247cfbc2: Status 404 returned error can't find the container with id 5c689ca017aeeaaf3423c772c60cabd0259bbbcb53e7373276c97665247cfbc2
Nov 03 12:43:19.553646 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.553552    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7590-29aa51"]
Nov 03 12:43:19.684349 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:19.684317    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b207d93d-b8b0-4d44-9311-9aae29ae8f44" path="/var/lib/kubelet/pods/b207d93d-b8b0-4d44-9311-9aae29ae8f44/volumes"
Nov 03 12:43:20.277363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:20.277327    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-29aa51" event={"ID":"e697b731-d027-4803-b0b8-57f525d0e2c1","Type":"ContainerStarted","Data":"65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3"}
Nov 03 12:43:20.277816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:20.277371    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7590-29aa51" event={"ID":"e697b731-d027-4803-b0b8-57f525d0e2c1","Type":"ContainerStarted","Data":"5c689ca017aeeaaf3423c772c60cabd0259bbbcb53e7373276c97665247cfbc2"}
Nov 03 12:43:20.381138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:20.381081    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7590-29aa51" podStartSLOduration=2.38106625 podStartE2EDuration="2.38106625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:20.37970789 +0000 UTC m=+3106.821910213" watchObservedRunningTime="2026-11-03 12:43:20.38106625 +0000 UTC m=+3106.823268555"
Nov 03 12:43:20.856758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:20.856724    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:21.469876 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.469846    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:21.613724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.613645    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\" (UniqueName: \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\") pod \"45608022-d8cf-452e-b602-17077ae8bee9\" (UID: \"45608022-d8cf-452e-b602-17077ae8bee9\") "
Nov 03 12:43:21.615674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.615644    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2" pod "45608022-d8cf-452e-b602-17077ae8bee9" (UID: "45608022-d8cf-452e-b602-17077ae8bee9"). InnerVolumeSpecName "kube-api-access-ms7s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:21.681563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.681529    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45608022-d8cf-452e-b602-17077ae8bee9" path="/var/lib/kubelet/pods/45608022-d8cf-452e-b602-17077ae8bee9/volumes"
Nov 03 12:43:21.715535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.715508    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7s2\" (UniqueName: \"kubernetes.io/projected/45608022-d8cf-452e-b602-17077ae8bee9-kube-api-access-ms7s2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:21.792813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.792780    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7590-29aa51"]
Nov 03 12:43:21.800503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:21.800476    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7590-29aa51"]
Nov 03 12:43:22.132120 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.132082    5784 patch_prober.go:29] interesting pod/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:43:22.132292 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.132147    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:43:22.295488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.295457    5784 generic.go:356] "Generic (PLEG): container finished" podID="45608022-d8cf-452e-b602-17077ae8bee9" containerID="cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae" exitCode=137
Nov 03 12:43:22.295674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.295510    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-b3c89f"
Nov 03 12:43:22.295674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.295557    5784 scope.go:122] "RemoveContainer" containerID="cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae"
Nov 03 12:43:22.295815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.295788    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7590-29aa51" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" containerName="test-curl-7590-29aa51" containerID="cri-o://65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3" gracePeriod=2
Nov 03 12:43:22.305126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.305104    5784 scope.go:122] "RemoveContainer" containerID="cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae"
Nov 03 12:43:22.305417 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:22.305393    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae\": container with ID starting with cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae not found: ID does not exist" containerID="cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae"
Nov 03 12:43:22.305516 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:22.305431    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae"} err="failed to get container status \"cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae\": rpc error: code = NotFound desc = could not find container \"cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae\": container with ID starting with cc4594641ecd03a1d80622a4b067fb3c8fbf1cd1206fdd38acf9912119b850ae not found: ID does not exist"
Nov 03 12:43:22.342993 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:22.342959    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-29aa51\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:22.390086 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:22.389997    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-29aa51\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:23.305122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.305088    5784 generic.go:356] "Generic (PLEG): container finished" podID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerID="684288b7f72de52788d8eaf7e39cdcd9b032ae7dc1331ee3389179dbd2c5aaaa" exitCode=0
Nov 03 12:43:23.305525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.305136    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" event={"ID":"bc878c14-eae2-4e23-b732-ffbff8a28bca","Type":"ContainerDied","Data":"684288b7f72de52788d8eaf7e39cdcd9b032ae7dc1331ee3389179dbd2c5aaaa"}
Nov 03 12:43:23.389113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.389086    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:43:23.431137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.431104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:23.538700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538664    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.538871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538779    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.538871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538821    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.538871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538862    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538915    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.538984    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539135 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539028    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539135 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539076    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539135 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539128    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\") pod \"bc878c14-eae2-4e23-b732-ffbff8a28bca\" (UID: \"bc878c14-eae2-4e23-b732-ffbff8a28bca\") "
Nov 03 12:43:23.539293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539132    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:23.539293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539146    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:23.539594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539455    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:43:23.539594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539566    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:23.539594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539579    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc878c14-eae2-4e23-b732-ffbff8a28bca-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.539819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539598    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.539819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539678    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:23.539819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.539711    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.542114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.541428    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:23.542205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.541586    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "kube-api-access-jxs4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:23.542205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.541804    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:43:23.542205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.541865    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo" pod "bc878c14-eae2-4e23-b732-ffbff8a28bca" (UID: "bc878c14-eae2-4e23-b732-ffbff8a28bca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:43:23.588854 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:23.588816    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-29aa51\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:23.641119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641083    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.641119 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641122    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.641335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641138    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxs4c\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-kube-api-access-jxs4c\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.641335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641156    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.641335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641170    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.641335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:23.641183    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc878c14-eae2-4e23-b732-ffbff8a28bca-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:23.726179 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:23.726131    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-29aa51\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:24.193401 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.192723    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:24.314795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.314718    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"
Nov 03 12:43:24.314795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.314727    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28" event={"ID":"bc878c14-eae2-4e23-b732-ffbff8a28bca","Type":"ContainerDied","Data":"aa08e5067c62c7b064f2cdd161f64e672b4ac1f6900e4c01353c930afff4c7f7"}
Nov 03 12:43:24.315268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.314971    5784 scope.go:122] "RemoveContainer" containerID="684288b7f72de52788d8eaf7e39cdcd9b032ae7dc1331ee3389179dbd2c5aaaa"
Nov 03 12:43:24.490658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.490537    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:43:24.493215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.493183    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9893f0-openshift-default-54fcb5b8c5-jch28"]
Nov 03 12:43:24.518454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.518419    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:24.578526 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.578499    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:24.656244 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.655740    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\" (UniqueName: \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\") pod \"e697b731-d027-4803-b0b8-57f525d0e2c1\" (UID: \"e697b731-d027-4803-b0b8-57f525d0e2c1\") "
Nov 03 12:43:24.658527 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.658500    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh" pod "e697b731-d027-4803-b0b8-57f525d0e2c1" (UID: "e697b731-d027-4803-b0b8-57f525d0e2c1"). InnerVolumeSpecName "kube-api-access-c6nlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:43:24.757072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:24.756990    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nlh\" (UniqueName: \"kubernetes.io/projected/e697b731-d027-4803-b0b8-57f525d0e2c1-kube-api-access-c6nlh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:43:25.323289 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.323256    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:25.326168 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.326138    5784 generic.go:356] "Generic (PLEG): container finished" podID="e697b731-d027-4803-b0b8-57f525d0e2c1" containerID="65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3" exitCode=137
Nov 03 12:43:25.326342 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.326184    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:25.326342 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.326258    5784 scope.go:122] "RemoveContainer" containerID="65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3"
Nov 03 12:43:25.339742 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.339714    5784 scope.go:122] "RemoveContainer" containerID="65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3"
Nov 03 12:43:25.340062 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:25.340036    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3\": container with ID starting with 65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3 not found: ID does not exist" containerID="65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3"
Nov 03 12:43:25.340147 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.340073    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3"} err="failed to get container status \"65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3\": rpc error: code = NotFound desc = could not find container \"65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3\": container with ID starting with 65647a7dc6d75b256cf9d85fd64d4e6c56d4d1f8232d099bb9483fb99a7c79e3 not found: ID does not exist"
Nov 03 12:43:25.381887 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:25.381842    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7590-29aa51\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" pod="opendatahub/test-curl-7590-29aa51"
Nov 03 12:43:25.681272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.681236    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" path="/var/lib/kubelet/pods/bc878c14-eae2-4e23-b732-ffbff8a28bca/volumes"
Nov 03 12:43:25.681680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:25.681664    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" path="/var/lib/kubelet/pods/e697b731-d027-4803-b0b8-57f525d0e2c1/volumes"
Nov 03 12:43:26.475356 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:26.475321    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:43:29.729066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:29.729035    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:29.913870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:29.913830    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:30.276151 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:30.276116    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:31.677301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:31.677268    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:31.843426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:31.843361    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:32.052325 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.052249    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 12:43:32.052858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.052836    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy"
Nov 03 12:43:32.052858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.052855    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy"
Nov 03 12:43:32.053095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.052870    5784 cpu_manager.go:414] "containerMap: removing container" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" containerName="test-curl-7590-29aa51"
Nov 03 12:43:32.053095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.052877    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" containerName="test-curl-7590-29aa51"
Nov 03 12:43:32.053095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.053053    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc878c14-eae2-4e23-b732-ffbff8a28bca" containerName="istio-proxy"
Nov 03 12:43:32.053095 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.053068    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e697b731-d027-4803-b0b8-57f525d0e2c1" containerName="test-curl-7590-29aa51"
Nov 03 12:43:32.056629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.056585    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.143133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.143101    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.143304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.143273    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmw47\" (UniqueName: \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.151287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.151255    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-02f899\""
Nov 03 12:43:32.151559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.151369    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-02f899-dockercfg-vcdh8\""
Nov 03 12:43:32.155303 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.155259    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 12:43:32.217922 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.217886    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 12:43:32.221978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.221955    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.245089 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.245059    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.245248 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.245169    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmw47\" (UniqueName: \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.245741 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.245719    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.274044 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.274018    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 12:43:32.301160 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.301132    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmw47\" (UniqueName: \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\") pod \"payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.345966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.345873    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88sq\" (UniqueName: \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.345966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.345950    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.371456 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.371423    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:32.449911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.447949    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.449911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.448167    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88sq\" (UniqueName: \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.449911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.449478    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.501293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.501261    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88sq\" (UniqueName: \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\") pod \"payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.536578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.536549    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:32.739432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.739403    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 12:43:32.741789 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:32.741760    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa2ccfbf_b68c_44a7_8e2c_6384fbe2a565.slice/crio-bcec64174798ec302491e09a6428aa628459024a9a778f9afb89080014a25bbf WatchSource:0}: Error finding container bcec64174798ec302491e09a6428aa628459024a9a778f9afb89080014a25bbf: Status 404 returned error can't find the container with id bcec64174798ec302491e09a6428aa628459024a9a778f9afb89080014a25bbf
Nov 03 12:43:32.904554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:32.904527    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 12:43:32.905245 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:32.905219    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1518f5d4_a5dc_4141_a297_e52ad0cf5d80.slice/crio-fa9dc357f5184dfe94e9586c8ac6d0694a74873d5424ae21e5cb732f7801de9f WatchSource:0}: Error finding container fa9dc357f5184dfe94e9586c8ac6d0694a74873d5424ae21e5cb732f7801de9f: Status 404 returned error can't find the container with id fa9dc357f5184dfe94e9586c8ac6d0694a74873d5424ae21e5cb732f7801de9f
Nov 03 12:43:33.384670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.384635    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" event={"ID":"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565","Type":"ContainerStarted","Data":"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"}
Nov 03 12:43:33.384866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.384693    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" event={"ID":"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565","Type":"ContainerStarted","Data":"bcec64174798ec302491e09a6428aa628459024a9a778f9afb89080014a25bbf"}
Nov 03 12:43:33.384866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.384726    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:33.386426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.386398    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" event={"ID":"1518f5d4-a5dc-4141-a297-e52ad0cf5d80","Type":"ContainerStarted","Data":"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"}
Nov 03 12:43:33.386536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.386453    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:33.386536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.386476    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" event={"ID":"1518f5d4-a5dc-4141-a297-e52ad0cf5d80","Type":"ContainerStarted","Data":"fa9dc357f5184dfe94e9586c8ac6d0694a74873d5424ae21e5cb732f7801de9f"}
Nov 03 12:43:33.491438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.491372    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" podStartSLOduration=1.491357454 podStartE2EDuration="1.491357454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:33.489296637 +0000 UTC m=+3119.931498969" watchObservedRunningTime="2026-11-03 12:43:33.491357454 +0000 UTC m=+3119.933559762"
Nov 03 12:43:33.595067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:33.595016    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" podStartSLOduration=1.595002303 podStartE2EDuration="1.595002303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:33.593050141 +0000 UTC m=+3120.035252473" watchObservedRunningTime="2026-11-03 12:43:33.595002303 +0000 UTC m=+3120.037204607"
Nov 03 12:43:35.121112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:35.121010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:40.015557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:40.015481    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:44.248090 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.248052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:44.396365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.396339    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 12:43:44.396542 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.396422    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 12:43:44.493865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.493830    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 12:43:44.498882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.498829    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.542098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.542067    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 12:43:44.546726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.546699    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a4489c-serving-cert\""
Nov 03 12:43:44.579593 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.579554    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.579758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.579610    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.579758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.579720    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.579846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.579768    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qj5\" (UniqueName: \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.680536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.680465    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.680714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.680503    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:44.680714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.680567    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.680714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.680659    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.680907 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:44.680725    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-serving-cert: secret "maas-api-e2e-shared-a-w1-a4489c-serving-cert" not found
Nov 03 12:43:44.680907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.680724    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qj5\" (UniqueName: \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.680907 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:44.680787    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls podName:382baf3a-d92f-4238-813c-01fcd9b0cc69 nodeName:}" failed. No retries permitted until 2026-11-03 12:43:45.180766427 +0000 UTC m=+3131.622968724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls") pod "maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" (UID: "382baf3a-d92f-4238-813c-01fcd9b0cc69") : secret "maas-api-e2e-shared-a-w1-a4489c-serving-cert" not found
Nov 03 12:43:44.681138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.681114    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.683226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.683203    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.734023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.733985    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qj5\" (UniqueName: \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:44.865523 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:44.865447    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:45.185708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:45.185668    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:45.188341 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:45.188312    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:45.412818 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:45.412776    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:45.801611 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:45.801578    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 12:43:45.802826 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:43:45.802793    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382baf3a_d92f_4238_813c_01fcd9b0cc69.slice/crio-0c155446b1636398ef1a7da98e74657d3b2eacf79f7548ed0cdb17f64135c536 WatchSource:0}: Error finding container 0c155446b1636398ef1a7da98e74657d3b2eacf79f7548ed0cdb17f64135c536: Status 404 returned error can't find the container with id 0c155446b1636398ef1a7da98e74657d3b2eacf79f7548ed0cdb17f64135c536
Nov 03 12:43:46.475285 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:46.475243    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" event={"ID":"382baf3a-d92f-4238-813c-01fcd9b0cc69","Type":"ContainerStarted","Data":"0c155446b1636398ef1a7da98e74657d3b2eacf79f7548ed0cdb17f64135c536"}
Nov 03 12:43:47.483758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:47.483719    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" event={"ID":"382baf3a-d92f-4238-813c-01fcd9b0cc69","Type":"ContainerStarted","Data":"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"}
Nov 03 12:43:47.484191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:47.483932    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:47.593104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:47.593041    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" podStartSLOduration=2.691808076 podStartE2EDuration="3.593025669s" totalImagesPullingTime="901.217593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:43:47.590424345 +0000 UTC m=+3134.032626675" watchObservedRunningTime="2026-11-03 12:43:47.593025669 +0000 UTC m=+3134.035228003"
Nov 03 12:43:49.209229 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:49.209190    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:53.496682 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:53.496652    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 12:43:53.612367 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:53.612332    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:56.473491 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:43:56.473450    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:43:57.738854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:57.738821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:43:58.074543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:43:58.074441    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:02.507466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:02.507428    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:06.346819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:06.346778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:06.919724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:06.919686    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:08.504647 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.504615    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:44:08.509090 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.509070    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.562126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.562094    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:44:08.584687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.584650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:08.603909 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.603884    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52ab15\"/\"kube-root-ca.crt\""
Nov 03 12:44:08.604070 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.604024    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52ab15\"/\"default-dockercfg-w54nz\""
Nov 03 12:44:08.604134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.604078    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52ab15\"/\"openshift-service-ca.crt\""
Nov 03 12:44:08.604204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.604183    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52ab15\"/\"explicit-model-52ab15-kserve-self-signed-certs\""
Nov 03 12:44:08.637519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637483    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.637720 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637624    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.637720 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637687    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.637842 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637723    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.637899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637839    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.637949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.637896    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vppnn\" (UniqueName: \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.738927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.738893    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.738954    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739009    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vppnn\" (UniqueName: \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739069    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739122 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739123    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739184    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739558    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739762 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739646    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.739918 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.739892    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.742145 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.742121    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.747858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.747835    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.792233 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.792146    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vppnn\" (UniqueName: \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\") pod \"explicit-model-52ab15-kserve-6cfcf9b567-tnkk7\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") " pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:08.822223 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:08.822150    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:09.213952 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:09.213924    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:44:09.214771 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:44:09.214744    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7ac10b_b5b8_4f75_8e16_20c4378ed9f2.slice/crio-c21d5b9d404ff2b6212d3065ab8437235c805f8595407d03c7007144ad6a8a57 WatchSource:0}: Error finding container c21d5b9d404ff2b6212d3065ab8437235c805f8595407d03c7007144ad6a8a57: Status 404 returned error can't find the container with id c21d5b9d404ff2b6212d3065ab8437235c805f8595407d03c7007144ad6a8a57
Nov 03 12:44:09.683994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:09.683959    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" event={"ID":"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2","Type":"ContainerStarted","Data":"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"}
Nov 03 12:44:09.684394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:09.684006    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" event={"ID":"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2","Type":"ContainerStarted","Data":"c21d5b9d404ff2b6212d3065ab8437235c805f8595407d03c7007144ad6a8a57"}
Nov 03 12:44:09.786302 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:09.786251    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" podStartSLOduration=1.7862374619999999 podStartE2EDuration="1.786237462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:09.784080357 +0000 UTC m=+3156.226282682" watchObservedRunningTime="2026-11-03 12:44:09.786237462 +0000 UTC m=+3156.228439767"
Nov 03 12:44:09.822936 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:09.822904    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:10.836754 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:10.836710    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:11.333783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:11.333753    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:15.723272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:15.723239    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:18.823331 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:18.823294    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:18.835559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:18.835533    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:19.759639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:19.759606    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:44:20.223306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:20.223274    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:24.469003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:24.468969    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:24.623775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:24.623738    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:25.174790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:25.174753    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:25.463934 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:25.463801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:26.045243 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:26.045198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:26.474393 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:44:26.474339    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:44:29.054836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:29.054795    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:29.724693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:29.724545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:29.892985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:29.892943    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:30.178158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:30.178114    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:32.804522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:32.804492    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:32.972678 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:32.972644    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:33.661302 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.661270    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 12:44:33.665228 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.665208    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.728799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.728769    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.728961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.728885    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.728961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.728928    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.729050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.729024    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mfv\" (UniqueName: \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.761756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.761728    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-02f899-serving-cert\""
Nov 03 12:44:33.766360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.766329    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 12:44:33.830301 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.830269    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.830791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.830391    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mfv\" (UniqueName: \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.830791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.830631    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.830791 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:44:33.830700    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-serving-cert: secret "maas-api-e2e-extmod-a-02f899-serving-cert" not found
Nov 03 12:44:33.830974 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.830803    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.830974 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:44:33.830816    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls podName:9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795 nodeName:}" failed. No retries permitted until 2026-11-03 12:44:34.330792987 +0000 UTC m=+3180.772995269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls") pod "maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" (UID: "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795") : secret "maas-api-e2e-extmod-a-02f899-serving-cert" not found
Nov 03 12:44:33.831070 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.831057    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.833092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.833070    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:33.870783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.870757    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:33.884480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:33.884452    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mfv\" (UniqueName: \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:34.027186 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.027074    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:34.178450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.178405    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:34.337531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.337448    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:34.339919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.339887    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") pod \"maas-api-e2e-extmod-a-02f899-66bdc69656-z497t\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:34.580486 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.580453    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:34.945408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:34.945359    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 12:44:34.946310 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:44:34.946284    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e717b5e_fbc6_4aa3_9ec0_33d35e2d1795.slice/crio-d34e19b4f2ab8a8d3ac063725900b9c5ddd1e4fe674020dc62cbf98bad805287 WatchSource:0}: Error finding container d34e19b4f2ab8a8d3ac063725900b9c5ddd1e4fe674020dc62cbf98bad805287: Status 404 returned error can't find the container with id d34e19b4f2ab8a8d3ac063725900b9c5ddd1e4fe674020dc62cbf98bad805287
Nov 03 12:44:35.862661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:35.862629    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" event={"ID":"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795","Type":"ContainerStarted","Data":"d34e19b4f2ab8a8d3ac063725900b9c5ddd1e4fe674020dc62cbf98bad805287"}
Nov 03 12:44:36.870875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:36.870831    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" event={"ID":"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795","Type":"ContainerStarted","Data":"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"}
Nov 03 12:44:36.871410 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:36.870981    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:36.978049 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:36.977998    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" podStartSLOduration=3.112650613 podStartE2EDuration="3.977984141s" totalImagesPullingTime="865.333528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:44:36.976917853 +0000 UTC m=+3183.419120182" watchObservedRunningTime="2026-11-03 12:44:36.977984141 +0000 UTC m=+3183.420186447"
Nov 03 12:44:38.425148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:38.425117    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:42.716520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:42.716483    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:42.879239 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:42.879205    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:42.883629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:42.883605    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 12:44:43.296932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:43.296892    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:43.564888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:43.564802    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:45.082827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:45.082788    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:44:45.112791 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:45.112761    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:45.577801 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:45.577766    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:47.102666 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:47.102624    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:47.340372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:47.340338    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:47.524843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:47.524810    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:47.860544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:47.860424    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:50.604701 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:50.604665    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:52.036885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:52.036850    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:52.258966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:52.258922    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:52.412368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:52.412326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:56.474319 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:44:56.474281    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:44:56.714220 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:56.714185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:44:58.052099 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:44:58.052058    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:00.103097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.103028    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" podUID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" containerName="main" containerID="cri-o://4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52" gracePeriod=45
Nov 03 12:45:00.384255 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.384220    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:45:00.538998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.538961    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.539212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539015    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.539212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539172    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.539212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539201    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539252    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\" (UniqueName: \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539285    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\") pod \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\" (UID: \"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2\") "
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539616    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539852    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539914    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:00.540365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.539925    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:00.541994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.541971    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:45:00.542129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.542108    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:45:00.542310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.542287    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn" pod "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" (UID: "cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2"). InnerVolumeSpecName "kube-api-access-vppnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:00.641265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.641230    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:00.641265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.641263    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:00.641543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.641278    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:00.641543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.641292    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vppnn\" (UniqueName: \"kubernetes.io/projected/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-kube-api-access-vppnn\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:00.641543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:00.641331    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:01.046188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.046097    5784 generic.go:356] "Generic (PLEG): container finished" podID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" containerID="4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52" exitCode=0
Nov 03 12:45:01.046188 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.046145    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"
Nov 03 12:45:01.046434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.046187    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" event={"ID":"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2","Type":"ContainerDied","Data":"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"}
Nov 03 12:45:01.046434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.046239    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7" event={"ID":"cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2","Type":"ContainerDied","Data":"c21d5b9d404ff2b6212d3065ab8437235c805f8595407d03c7007144ad6a8a57"}
Nov 03 12:45:01.046434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.046262    5784 scope.go:122] "RemoveContainer" containerID="4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"
Nov 03 12:45:01.059504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.059477    5784 scope.go:122] "RemoveContainer" containerID="4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"
Nov 03 12:45:01.059811 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:01.059790    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52\": container with ID starting with 4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52 not found: ID does not exist" containerID="4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"
Nov 03 12:45:01.059889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.059819    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52"} err="failed to get container status \"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52\": rpc error: code = NotFound desc = could not find container \"4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52\": container with ID starting with 4dd21d181664f40d4a55b1597fc9b0957fe8f8b1f8956c19939b54e9c9316b52 not found: ID does not exist"
Nov 03 12:45:01.072752 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.072726    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:01.198844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.198806    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:45:01.244998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.244968    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-52ab15/explicit-model-52ab15-kserve-6cfcf9b567-tnkk7"]
Nov 03 12:45:01.430843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.430805    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:01.580737 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.580699    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:01.681985 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:01.681906    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" path="/var/lib/kubelet/pods/cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2/volumes"
Nov 03 12:45:02.103038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:02.102956    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:02.529179 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:02.529142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:05.895789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:05.895757    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:07.269783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:07.269750    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:08.545069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:08.545029    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:10.699583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:10.699551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:10.931547 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:10.931514    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:11.094334 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:11.094240    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:15.442805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:15.442758    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:19.846777 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:19.846744    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:24.253419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:24.253368    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:26.474091 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:26.474048    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:45:27.432555 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:27.432515    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:28.389653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:28.389620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:28.544674 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:28.544641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:28.714063 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:28.713990    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:33.000662 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:33.000620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:33.392979 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:33.392949    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:37.988082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:37.988046    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:45:37.988667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:37.988651    5784 cpu_manager.go:414] "containerMap: removing container" podUID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" containerName="main"
Nov 03 12:45:37.988727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:37.988669    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" containerName="main"
Nov 03 12:45:37.988798 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:37.988786    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7ac10b-b5b8-4f75-8e16-20c4378ed9f2" containerName="main"
Nov 03 12:45:37.993496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:37.993476    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.042249 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.042210    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bebc6d-serving-cert\""
Nov 03 12:45:38.045235 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.045207    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:45:38.159859 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.159826    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.160024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.159889    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.160024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.159961    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2kck\" (UniqueName: \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.160024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.160017    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.193853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.193817    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:38.261011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.260924    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.261011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.260981    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.261011 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.261006    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2kck\" (UniqueName: \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.261297 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.261045    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.261297 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:38.261076    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-serving-cert: secret "maas-api-e2e-extmod-a-bebc6d-serving-cert" not found
Nov 03 12:45:38.261297 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:38.261152    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls podName:23a38c0b-dcae-46d7-9845-fb655db2fba0 nodeName:}" failed. No retries permitted until 2026-11-03 12:45:38.761130545 +0000 UTC m=+3245.203332830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls") pod "maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" (UID: "23a38c0b-dcae-46d7-9845-fb655db2fba0") : secret "maas-api-e2e-extmod-a-bebc6d-serving-cert" not found
Nov 03 12:45:38.261705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.261678    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.263800 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.263767    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.314704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.314674    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2kck\" (UniqueName: \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.354116 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.354090    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:38.511088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.511014    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:38.766977 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.766891    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.769615 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.769584    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:38.906786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:38.906755    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:39.274671 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:39.274633    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:45:39.275111 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:45:39.274686    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a38c0b_dcae_46d7_9845_fb655db2fba0.slice/crio-bed5b8359e28177558b034774e10a893e30161b8ef65c328b3006a263702cfb8 WatchSource:0}: Error finding container bed5b8359e28177558b034774e10a893e30161b8ef65c328b3006a263702cfb8: Status 404 returned error can't find the container with id bed5b8359e28177558b034774e10a893e30161b8ef65c328b3006a263702cfb8
Nov 03 12:45:39.323832 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:39.323797    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" event={"ID":"23a38c0b-dcae-46d7-9845-fb655db2fba0","Type":"ContainerStarted","Data":"bed5b8359e28177558b034774e10a893e30161b8ef65c328b3006a263702cfb8"}
Nov 03 12:45:40.331770 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:40.331736    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" event={"ID":"23a38c0b-dcae-46d7-9845-fb655db2fba0","Type":"ContainerStarted","Data":"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"}
Nov 03 12:45:40.332178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:40.331812    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:40.450610 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:40.450554    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" podStartSLOduration=2.507020308 podStartE2EDuration="3.450540363s" totalImagesPullingTime="943.520055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:45:40.449160549 +0000 UTC m=+3246.891362873" watchObservedRunningTime="2026-11-03 12:45:40.450540363 +0000 UTC m=+3246.892742701"
Nov 03 12:45:42.941369 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:42.941289    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:46.344626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:46.344582    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:45:47.413428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:47.413396    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:49.867281 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:49.867234    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:50.022638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:50.022606    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:51.818137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:51.818100    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:55.839947 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:55.839880    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:45:55.840475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:55.840151    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" podUID="c9052b6b-ccb4-47d6-a721-b646fa715791" containerName="maas-api" containerID="cri-o://54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1" gracePeriod=30
Nov 03 12:45:55.958533 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:55.958493    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:56.124826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.124796    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:45:56.153661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.152665    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\" (UniqueName: \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\") pod \"c9052b6b-ccb4-47d6-a721-b646fa715791\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") "
Nov 03 12:45:56.153661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.152795    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\") pod \"c9052b6b-ccb4-47d6-a721-b646fa715791\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") "
Nov 03 12:45:56.153661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.152966    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\") pod \"c9052b6b-ccb4-47d6-a721-b646fa715791\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") "
Nov 03 12:45:56.153661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.153008    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") pod \"c9052b6b-ccb4-47d6-a721-b646fa715791\" (UID: \"c9052b6b-ccb4-47d6-a721-b646fa715791\") "
Nov 03 12:45:56.153661 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.153408    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca" pod "c9052b6b-ccb4-47d6-a721-b646fa715791" (UID: "c9052b6b-ccb4-47d6-a721-b646fa715791"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:45:56.157048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.157003    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls" pod "c9052b6b-ccb4-47d6-a721-b646fa715791" (UID: "c9052b6b-ccb4-47d6-a721-b646fa715791"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:45:56.157205 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.157112    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf" pod "c9052b6b-ccb4-47d6-a721-b646fa715791" (UID: "c9052b6b-ccb4-47d6-a721-b646fa715791"). InnerVolumeSpecName "kube-api-access-h47wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:45:56.158776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.158750    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs" pod "c9052b6b-ccb4-47d6-a721-b646fa715791" (UID: "c9052b6b-ccb4-47d6-a721-b646fa715791"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:45:56.254507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.254461    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9052b6b-ccb4-47d6-a721-b646fa715791-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:56.254507 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.254504    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:56.254775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.254520    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9052b6b-ccb4-47d6-a721-b646fa715791-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:56.254775 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.254534    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h47wf\" (UniqueName: \"kubernetes.io/projected/c9052b6b-ccb4-47d6-a721-b646fa715791-kube-api-access-h47wf\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:45:56.443517 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.443484    5784 generic.go:356] "Generic (PLEG): container finished" podID="c9052b6b-ccb4-47d6-a721-b646fa715791" containerID="54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1" exitCode=0
Nov 03 12:45:56.443685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.443531    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"
Nov 03 12:45:56.443685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.443558    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" event={"ID":"c9052b6b-ccb4-47d6-a721-b646fa715791","Type":"ContainerDied","Data":"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"}
Nov 03 12:45:56.443685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.443600    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q" event={"ID":"c9052b6b-ccb4-47d6-a721-b646fa715791","Type":"ContainerDied","Data":"bb25279b8ea248f2815d3f10bd285eedea9428fe0e0c12208646c7d63d3f8b88"}
Nov 03 12:45:56.443685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.443620    5784 scope.go:122] "RemoveContainer" containerID="54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"
Nov 03 12:45:56.455218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.455199    5784 scope.go:122] "RemoveContainer" containerID="54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"
Nov 03 12:45:56.455562 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:56.455536    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1\": container with ID starting with 54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1 not found: ID does not exist" containerID="54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"
Nov 03 12:45:56.455635 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.455573    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1"} err="failed to get container status \"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1\": rpc error: code = NotFound desc = could not find container \"54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1\": container with ID starting with 54bce9719007c33741bd1e1e7b4d3be906745f6e84fef106825398d7bbe73ba1 not found: ID does not exist"
Nov 03 12:45:56.471181 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:56.471154    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:45:56.596194 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.596137    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:45:56.641999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.641936    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52ab15-5ff789d5d9-pbf7q"]
Nov 03 12:45:56.877919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:56.877893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:57.459706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.459670    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:45:57.460363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.460340    5784 cpu_manager.go:414] "containerMap: removing container" podUID="c9052b6b-ccb4-47d6-a721-b646fa715791" containerName="maas-api"
Nov 03 12:45:57.460363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.460361    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9052b6b-ccb4-47d6-a721-b646fa715791" containerName="maas-api"
Nov 03 12:45:57.460585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.460576    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9052b6b-ccb4-47d6-a721-b646fa715791" containerName="maas-api"
Nov 03 12:45:57.465503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.465483    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.517196 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.517164    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:45:57.558745 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.558718    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-fd7881-serving-cert\""
Nov 03 12:45:57.565818 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.565793    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rpfs\" (UniqueName: \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.565961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.565850    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.565961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.565943    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.566038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.565990    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.667631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.667590    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.667838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.667653    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.667838 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.667785    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rpfs\" (UniqueName: \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.667963 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.667853    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.668002 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:57.667986    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-serving-cert: secret "maas-api-e2e-extmod-a-fd7881-serving-cert" not found
Nov 03 12:45:57.668067 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:45:57.668055    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls podName:6c18e3c2-fc1c-4a0a-9157-63d95cdc857f nodeName:}" failed. No retries permitted until 2026-11-03 12:45:58.168034 +0000 UTC m=+3264.610236296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls") pod "maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" (UID: "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f") : secret "maas-api-e2e-extmod-a-fd7881-serving-cert" not found
Nov 03 12:45:57.668450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.668428    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.670386 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.670349    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.670665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.670641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:57.682557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.682530    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9052b6b-ccb4-47d6-a721-b646fa715791" path="/var/lib/kubelet/pods/c9052b6b-ccb4-47d6-a721-b646fa715791/volumes"
Nov 03 12:45:57.724596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.724521    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rpfs\" (UniqueName: \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:57.827602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.827572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:57.995461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:57.995342    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:58.174665 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.174619    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:58.177054 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.177031    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:58.210364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.210311    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:58.367723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.367641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:58.380672 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.380645    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:45:58.774143 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.774113    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:45:58.808504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:58.808464    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:59.206764 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:59.206733    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:45:59.471058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:45:59.470967    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" event={"ID":"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f","Type":"ContainerStarted","Data":"cab8027bdbe5fc51c87855ea02d43d9b7882de23698fb96b5aa9ed4cdefd682c"}
Nov 03 12:46:00.479807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:00.479766    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" event={"ID":"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f","Type":"ContainerStarted","Data":"f49cd1f187dba488525a050fe5355772ff36e53bc42d7bbe2e45980700e9e96b"}
Nov 03 12:46:00.480305 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:00.479861    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:46:00.586865 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:00.586810    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" podStartSLOduration=2.678920407 podStartE2EDuration="3.586795679s" totalImagesPullingTime="907.875272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:00.585793801 +0000 UTC m=+3267.027996101" watchObservedRunningTime="2026-11-03 12:46:00.586795679 +0000 UTC m=+3267.028997983"
Nov 03 12:46:02.508114 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:02.508080    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:02.808691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:02.808585    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:02.973128 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:02.973091    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:03.133139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:03.133110    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:05.014154 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:05.014116    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:05.175242 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:05.175195    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:05.336584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:05.336505    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:05.885169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:05.885133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:06.036552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:06.036519    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:06.491705 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:06.491678    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:46:06.971067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:06.971033    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:07.257621 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:07.257520    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:07.442578 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:07.442544    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:08.089857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:08.089824    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:08.679256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:08.679220    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:09.063487 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:09.063393    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:10.649133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:10.649104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:10.840187 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:10.840153    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:11.044635 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:11.044549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:12.207950 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:12.207915    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:13.284219 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.284188    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:46:13.288723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.288688    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.383345 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.383296    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ff65b5\"/\"openshift-service-ca.crt\""
Nov 03 12:46:13.383529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.383321    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ff65b5\"/\"default-dockercfg-wwq68\""
Nov 03 12:46:13.383529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.383338    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-ff65b5\"/\"kube-root-ca.crt\""
Nov 03 12:46:13.383529 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.383409    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-ff65b5\"/\"noten-model-ff65b5-kserve-self-signed-certs\""
Nov 03 12:46:13.386320 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.386298    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:46:13.399053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.399026    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:13.453875 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.453844    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.454084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.453919    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7jt\" (UniqueName: \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.454084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.453950    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.454084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.454029    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.454268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.454155    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.454268 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.454213    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555307    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555358    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555416    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555457    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7jt\" (UniqueName: \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555492    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555728 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555530    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555890 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555867    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.555948 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.555920    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.556388 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.556349    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.562803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.562778    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.563197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.563136    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.608756 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.608710    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7jt\" (UniqueName: \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\") pod \"noten-model-ff65b5-kserve-6955c84677-2f8q2\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") " pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:13.901965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:13.901925    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:14.271081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:14.271040    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:46:14.271915 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:46:14.271886    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod900f8cb2_df9a_44ce_98eb_c40e415be040.slice/crio-557cbdab1b69847801581b4bcf7b9a28767348d173a94e62729fffa57f81ddea WatchSource:0}: Error finding container 557cbdab1b69847801581b4bcf7b9a28767348d173a94e62729fffa57f81ddea: Status 404 returned error can't find the container with id 557cbdab1b69847801581b4bcf7b9a28767348d173a94e62729fffa57f81ddea
Nov 03 12:46:14.579872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:14.579775    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" event={"ID":"900f8cb2-df9a-44ce-98eb-c40e415be040","Type":"ContainerStarted","Data":"6d1ea5fce5ac6cc4b02031ceb2b123234f9ed124d77322d9bf4ce22d3bf66d29"}
Nov 03 12:46:14.579872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:14.579826    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" event={"ID":"900f8cb2-df9a-44ce-98eb-c40e415be040","Type":"ContainerStarted","Data":"557cbdab1b69847801581b4bcf7b9a28767348d173a94e62729fffa57f81ddea"}
Nov 03 12:46:14.687438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:14.687360    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" podStartSLOduration=1.687342701 podStartE2EDuration="1.687342701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:14.684632492 +0000 UTC m=+3281.126834824" watchObservedRunningTime="2026-11-03 12:46:14.687342701 +0000 UTC m=+3281.129545006"
Nov 03 12:46:14.902965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:14.902916    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:16.630421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:16.630372    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:20.697999 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.697968    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 12:46:20.702291 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.702272    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.751885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.751855    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-6fb3a8-serving-cert\""
Nov 03 12:46:20.800543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.800503    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 12:46:20.844129 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.844080    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.844316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.844158    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwkb9\" (UniqueName: \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.844405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.844367    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.844494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.844473    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.906918 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.906879    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:20.945364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.945324    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.945724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.945440    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.945724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.945487    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.945724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.945534    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwkb9\" (UniqueName: \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.947536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.946559    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.947536 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:46:20.946622    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-serving-cert: secret "maas-api-e2e-shared-b-w1-6fb3a8-serving-cert" not found
Nov 03 12:46:20.947536 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:46:20.946702    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls podName:2d63a51c-bf03-4100-addc-0264a9058016 nodeName:}" failed. No retries permitted until 2026-11-03 12:46:21.446683275 +0000 UTC m=+3287.888885560 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls") pod "maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" (UID: "2d63a51c-bf03-4100-addc-0264a9058016") : secret "maas-api-e2e-shared-b-w1-6fb3a8-serving-cert" not found
Nov 03 12:46:20.949427 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.949330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:20.999163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:20.999131    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwkb9\" (UniqueName: \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:21.054869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.054834    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:21.451739 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.451698    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:21.454339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.454305    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:21.577601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.577569    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:21.616861 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.616821    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:21.992273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:21.992216    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 12:46:21.993640 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:46:21.993608    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d63a51c_bf03_4100_addc_0264a9058016.slice/crio-c52ce2f73240a11e0f767a00b1166e5ee337c2f934567b4c80f7e0573bd40abc WatchSource:0}: Error finding container c52ce2f73240a11e0f767a00b1166e5ee337c2f934567b4c80f7e0573bd40abc: Status 404 returned error can't find the container with id c52ce2f73240a11e0f767a00b1166e5ee337c2f934567b4c80f7e0573bd40abc
Nov 03 12:46:22.639417 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:22.639364    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" event={"ID":"2d63a51c-bf03-4100-addc-0264a9058016","Type":"ContainerStarted","Data":"c52ce2f73240a11e0f767a00b1166e5ee337c2f934567b4c80f7e0573bd40abc"}
Nov 03 12:46:23.648626 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:23.648572    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" event={"ID":"2d63a51c-bf03-4100-addc-0264a9058016","Type":"ContainerStarted","Data":"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"}
Nov 03 12:46:23.649150 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:23.648654    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:23.769428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:23.769350    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" podStartSLOduration=2.8936944430000002 podStartE2EDuration="3.769331598s" totalImagesPullingTime="875.637155ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:46:23.767369651 +0000 UTC m=+3290.209571958" watchObservedRunningTime="2026-11-03 12:46:23.769331598 +0000 UTC m=+3290.211533904"
Nov 03 12:46:23.902963 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:23.902882    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:23.915550 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:23.915526    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:24.671986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:24.671957    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:46:25.529029 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:25.528997    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:26.482184 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:46:26.482147    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:46:29.662823 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:29.662792    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 12:46:30.000305 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:30.000222    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:34.283935 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:34.283910    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:46:34.299645 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:34.299620    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:46:34.420494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:34.420449    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:38.767712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:38.767682    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:43.217636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:43.217586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:47.640350 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:47.640309    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:52.097867 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:52.097833    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:53.424997 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:53.424951    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:53.769726 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:53.769642    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:53.920434 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:53.920329    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:56.423211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:56.423178    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:56.472572 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:46:56.472537    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:46:58.710778 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:58.710739    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:46:58.834488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:58.834446    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:46:58.936335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:46:58.936299    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:00.798876 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:00.798846    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:02.768199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:02.767885    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:47:02.768199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:02.768115    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" containerID="cri-o://adcf9c00d19f0e0f2afa9e61c72271fc2265f9915eb51cd6bc4368aabefbd3e4" gracePeriod=30
Nov 03 12:47:02.974546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:02.974512    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:47:02.978443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:02.978418    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.004996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.004963    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005011    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005098    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005262 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005160    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8hnm\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005267    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005366    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005446    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005473    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.005629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.005606    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.026600 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.026523    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-6fb3a8-openshift-default-dockercfg-j9znp\""
Nov 03 12:47:03.058333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.058299    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:47:03.106877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.106842    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.106937    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.106957    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.106980    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107079 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107049    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107079    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107213    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107258    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107321 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107285    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8hnm\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107462    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107535    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107642    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107755    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.107833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.107794    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.109316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.109299    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.109567 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.109550    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.160094 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.160057    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.160265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.160167    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8hnm\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\") pod \"e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") " pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.276831 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.276745    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:03.293886 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.293853    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:03.711215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.711176    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:47:03.711862 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:47:03.711833    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c21463_38c8_4195_9211_d258590fe08f.slice/crio-c8bbcde0d2c06255a9477993c90a0f3cc94c41f0c4c8bfbbdb17b82b29779f1c WatchSource:0}: Error finding container c8bbcde0d2c06255a9477993c90a0f3cc94c41f0c4c8bfbbdb17b82b29779f1c: Status 404 returned error can't find the container with id c8bbcde0d2c06255a9477993c90a0f3cc94c41f0c4c8bfbbdb17b82b29779f1c
Nov 03 12:47:03.714294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.714259    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:47:03.714358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.714327    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:47:03.714421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.714357    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:47:03.941003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.940970    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" event={"ID":"f2c21463-38c8-4195-9211-d258590fe08f","Type":"ContainerStarted","Data":"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"}
Nov 03 12:47:03.941433 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:03.941024    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" event={"ID":"f2c21463-38c8-4195-9211-d258590fe08f","Type":"ContainerStarted","Data":"c8bbcde0d2c06255a9477993c90a0f3cc94c41f0c4c8bfbbdb17b82b29779f1c"}
Nov 03 12:47:04.050953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.050850    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podStartSLOduration=2.050835852 podStartE2EDuration="2.050835852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:04.050296143 +0000 UTC m=+3330.492498469" watchObservedRunningTime="2026-11-03 12:47:04.050835852 +0000 UTC m=+3330.493038202"
Nov 03 12:47:04.294804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.294761    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:04.296599 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.296568    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:04.296747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.296626    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:04.679595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.679562    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:04.995837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.995752    5784 patch_prober.go:29] interesting pod/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:47:04.995837 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:04.995815    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:47:05.294833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:05.294746    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:05.294833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:05.294810    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:05.366525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:05.366441    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:05.538344 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:05.538309    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:05.687820 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:05.687791    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:06.295203 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:06.295167    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:06.295636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:06.295223    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:07.294729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:07.294689    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:07.294926 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:07.294744    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:07.989405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:07.989140    5784 generic.go:356] "Generic (PLEG): container finished" podID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerID="adcf9c00d19f0e0f2afa9e61c72271fc2265f9915eb51cd6bc4368aabefbd3e4" exitCode=0
Nov 03 12:47:07.989405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:07.989319    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" event={"ID":"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d","Type":"ContainerDied","Data":"adcf9c00d19f0e0f2afa9e61c72271fc2265f9915eb51cd6bc4368aabefbd3e4"}
Nov 03 12:47:08.086813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.086783    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:47:08.190159 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190131    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190183    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190213    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190267    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190372 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190306    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190473    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190503    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190532    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:08.190652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190544    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190652 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190632    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:08.190927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190639    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\") pod \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\" (UID: \"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d\") "
Nov 03 12:47:08.190927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.190870    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:47:08.191091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191065    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:08.191215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191191    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:08.191300 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191281    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.191354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191307    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.191354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191324    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.191354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.191339    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.192633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.192606    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:47:08.193033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.193014    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "kube-api-access-nh4x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:47:08.193155 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.193135    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:47:08.193221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.193192    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy" pod "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" (UID: "c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:08.292425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.292372    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.292425 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.292424    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh4x8\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-kube-api-access-nh4x8\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.292722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.292441    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.292722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.292455    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.292722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.292470    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:08.295072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.295047    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:08.295139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.295102    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:08.999038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.999004    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"
Nov 03 12:47:08.999568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.999006    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9" event={"ID":"c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d","Type":"ContainerDied","Data":"21a6e7143d4127a49cb61fad9598a37b81380c67ace99b63705cbc60fedf7c56"}
Nov 03 12:47:08.999568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:08.999191    5784 scope.go:122] "RemoveContainer" containerID="adcf9c00d19f0e0f2afa9e61c72271fc2265f9915eb51cd6bc4368aabefbd3e4"
Nov 03 12:47:09.130461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.125046    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:09.200304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.200261    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:47:09.204637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.204612    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-91cd99-openshift-default-fb76578c9-jrnl9"]
Nov 03 12:47:09.294668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.294583    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Nov 03 12:47:09.294668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.294643    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Nov 03 12:47:09.392065 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.392035    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:09.682204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:09.682168    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" path="/var/lib/kubelet/pods/c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d/volumes"
Nov 03 12:47:10.299731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:10.299698    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:10.300107 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:10.300097    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:10.300841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:10.300823    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:47:10.601078 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:10.600993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:11.722896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:11.722821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:12.153653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:12.153625    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:12.310580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:12.310546    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:13.864391 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:13.864301    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" podUID="900f8cb2-df9a-44ce-98eb-c40e415be040" containerName="main" containerID="cri-o://6d1ea5fce5ac6cc4b02031ceb2b123234f9ed124d77322d9bf4ce22d3bf66d29" gracePeriod=45
Nov 03 12:47:14.042045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.041863    5784 generic.go:356] "Generic (PLEG): container finished" podID="900f8cb2-df9a-44ce-98eb-c40e415be040" containerID="6d1ea5fce5ac6cc4b02031ceb2b123234f9ed124d77322d9bf4ce22d3bf66d29" exitCode=0
Nov 03 12:47:14.042045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.041990    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" event={"ID":"900f8cb2-df9a-44ce-98eb-c40e415be040","Type":"ContainerDied","Data":"6d1ea5fce5ac6cc4b02031ceb2b123234f9ed124d77322d9bf4ce22d3bf66d29"}
Nov 03 12:47:14.143422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.143394    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:47:14.259045 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259003    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259118    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259152    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\" (UniqueName: \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259271    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\" (UniqueName: \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259308    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:14.259359 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259343    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259540 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259397    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\") pod \"900f8cb2-df9a-44ce-98eb-c40e415be040\" (UID: \"900f8cb2-df9a-44ce-98eb-c40e415be040\") "
Nov 03 12:47:14.259994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259782    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:14.259994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.259929    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:14.260241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.260215    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:14.262218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.262185    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "kube-api-access-wb7jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:47:14.263544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.263519    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:47:14.263814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.263791    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs" pod "900f8cb2-df9a-44ce-98eb-c40e415be040" (UID: "900f8cb2-df9a-44ce-98eb-c40e415be040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:47:14.360689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.360646    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:14.360689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.360681    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7jt\" (UniqueName: \"kubernetes.io/projected/900f8cb2-df9a-44ce-98eb-c40e415be040-kube-api-access-wb7jt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:14.360689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.360691    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/900f8cb2-df9a-44ce-98eb-c40e415be040-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:14.360689 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.360700    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:14.361070 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:14.360709    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/900f8cb2-df9a-44ce-98eb-c40e415be040-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:47:15.051485 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.051447    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2" event={"ID":"900f8cb2-df9a-44ce-98eb-c40e415be040","Type":"ContainerDied","Data":"557cbdab1b69847801581b4bcf7b9a28767348d173a94e62729fffa57f81ddea"}
Nov 03 12:47:15.052109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.051496    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"
Nov 03 12:47:15.052109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.051503    5784 scope.go:122] "RemoveContainer" containerID="6d1ea5fce5ac6cc4b02031ceb2b123234f9ed124d77322d9bf4ce22d3bf66d29"
Nov 03 12:47:15.106291 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.106256    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:15.204658 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.204624    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:47:15.210222 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.210196    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-ff65b5/noten-model-ff65b5-kserve-6955c84677-2f8q2"]
Nov 03 12:47:15.681662 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:15.681630    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900f8cb2-df9a-44ce-98eb-c40e415be040" path="/var/lib/kubelet/pods/900f8cb2-df9a-44ce-98eb-c40e415be040/volumes"
Nov 03 12:47:18.118552 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:18.118515    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:19.215435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.215400    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 12:47:19.216037 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216013    5784 cpu_manager.go:414] "containerMap: removing container" podUID="900f8cb2-df9a-44ce-98eb-c40e415be040" containerName="main"
Nov 03 12:47:19.216037 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216032    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900f8cb2-df9a-44ce-98eb-c40e415be040" containerName="main"
Nov 03 12:47:19.216191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216062    5784 cpu_manager.go:414] "containerMap: removing container" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy"
Nov 03 12:47:19.216191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216067    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy"
Nov 03 12:47:19.216191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216165    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="c696c1c7-0b7f-49f0-8aa5-c23f5467dd4d" containerName="istio-proxy"
Nov 03 12:47:19.216191 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.216178    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="900f8cb2-df9a-44ce-98eb-c40e415be040" containerName="main"
Nov 03 12:47:19.220785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.220766    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.315761 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.315725    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-94e247-serving-cert\""
Nov 03 12:47:19.318905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.318880    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 12:47:19.324075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.324050    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlkg\" (UniqueName: \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.324212 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.324119    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.324257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.324210    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.324257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.324250    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.418020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.417973    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:19.425310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.425280    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlkg\" (UniqueName: \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.425467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.425343    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.425467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.425408    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.425467 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.425443    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.425598 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:47:19.425471    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-serving-cert: secret "maas-api-e2e-extmod-b-94e247-serving-cert" not found
Nov 03 12:47:19.425598 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:47:19.425522    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls podName:6413238b-2d42-4ae0-ad5d-5e281f979903 nodeName:}" failed. No retries permitted until 2026-11-03 12:47:19.925508064 +0000 UTC m=+3346.367710364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls") pod "maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" (UID: "6413238b-2d42-4ae0-ad5d-5e281f979903") : secret "maas-api-e2e-extmod-b-94e247-serving-cert" not found
Nov 03 12:47:19.426115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.426094    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.427824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.427805    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.480640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.480570    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlkg\" (UniqueName: \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.572868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.572838    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:19.726549 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.726519    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:19.931565 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.931528    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:19.933956 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:19.933932    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") pod \"maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:20.134296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:20.134260    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:20.504853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:20.504824    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 12:47:20.505874 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:47:20.505847    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6413238b_2d42_4ae0_ad5d_5e281f979903.slice/crio-77e0c6a648fa7a25d4ca84f9c8708ec7dd8d953836b5846ee486055944c92d69 WatchSource:0}: Error finding container 77e0c6a648fa7a25d4ca84f9c8708ec7dd8d953836b5846ee486055944c92d69: Status 404 returned error can't find the container with id 77e0c6a648fa7a25d4ca84f9c8708ec7dd8d953836b5846ee486055944c92d69
Nov 03 12:47:21.093517 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:21.093485    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" event={"ID":"6413238b-2d42-4ae0-ad5d-5e281f979903","Type":"ContainerStarted","Data":"77e0c6a648fa7a25d4ca84f9c8708ec7dd8d953836b5846ee486055944c92d69"}
Nov 03 12:47:21.474632 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:21.474591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:22.101287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:22.101243    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" event={"ID":"6413238b-2d42-4ae0-ad5d-5e281f979903","Type":"ContainerStarted","Data":"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"}
Nov 03 12:47:22.101753 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:22.101308    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:22.210691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:22.210638    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" podStartSLOduration=2.223886661 podStartE2EDuration="3.210618655s" totalImagesPullingTime="986.731994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:47:22.207104009 +0000 UTC m=+3348.649306314" watchObservedRunningTime="2026-11-03 12:47:22.210618655 +0000 UTC m=+3348.652820960"
Nov 03 12:47:22.837318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:22.837286    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:23.426920 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:23.426884    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:23.693019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:23.692922    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:24.320879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:24.320838    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:26.475650 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:47:26.475618    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:47:27.267746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:27.267700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:27.437570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:27.437516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:27.680158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:27.680101    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:28.122225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:28.122143    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 12:47:29.210563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:29.210529    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:30.359227 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:30.359194    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:30.509257 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:30.509221    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:33.620667 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:33.620632    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:35.800640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:35.800604    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:37.476022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:37.475988    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:38.138265 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:38.138225    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:38.306556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:38.306517    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:38.524494 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:38.524395    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:42.644710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:42.644676    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:43.018118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:43.018034    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:43.179808 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:43.179776    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:43.355877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:43.355792    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:46.035983 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:46.035945    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:46.224907 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:46.224866    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:47.299029 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:47.298993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:51.116706 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:51.116667    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:52.009953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:52.009895    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:56.473924 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:47:56.473883    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:47:56.602272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:56.602243    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:47:59.045570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:47:59.045533    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:01.010200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:01.010167    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:05.393111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:05.393075    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:06.064677 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.064642    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:48:06.068843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.068823    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.163355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.163324    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"kube-root-ca.crt\""
Nov 03 12:48:06.163562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.163426    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"default-dockercfg-zl57l\""
Nov 03 12:48:06.163562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.163428    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"openshift-service-ca.crt\""
Nov 03 12:48:06.163562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.163480    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-a4489c\"/\"auth-test-model-a4489c-kserve-self-signed-certs\""
Nov 03 12:48:06.166139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166119    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166247    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166278 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166281    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166357    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166401    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166422    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jj7c\" (UniqueName: \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.166622 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.166501    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:48:06.195722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.195696    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:06.267005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.266964    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267005 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267010    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267027    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jj7c\" (UniqueName: \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267072    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267151    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267267 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267171    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267539 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267519    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267572    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.267604 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.267584    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.269277 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.269254    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.269696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.269675    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.320163 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.320087    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jj7c\" (UniqueName: \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\") pod \"auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") " pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.380743 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.380702    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:06.758448 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:06.758415    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:48:06.759673 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:48:06.759648    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fc8e96f_6e6a_46ef_b6de_2e1b28732e2a.slice/crio-4f5623993984274c253e68b1177172b71cecb4d54e3a54894fe41f4dd9762b03 WatchSource:0}: Error finding container 4f5623993984274c253e68b1177172b71cecb4d54e3a54894fe41f4dd9762b03: Status 404 returned error can't find the container with id 4f5623993984274c253e68b1177172b71cecb4d54e3a54894fe41f4dd9762b03
Nov 03 12:48:07.450961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:07.450923    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" event={"ID":"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a","Type":"ContainerStarted","Data":"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"}
Nov 03 12:48:07.451146 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:07.450973    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" event={"ID":"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a","Type":"ContainerStarted","Data":"4f5623993984274c253e68b1177172b71cecb4d54e3a54894fe41f4dd9762b03"}
Nov 03 12:48:07.557961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:07.557900    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" podStartSLOduration=1.557888147 podStartE2EDuration="1.557888147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:48:07.55534187 +0000 UTC m=+3393.997544200" watchObservedRunningTime="2026-11-03 12:48:07.557888147 +0000 UTC m=+3394.000090452"
Nov 03 12:48:08.381520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:08.381490    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:09.842432 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:09.842361    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:14.207214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:14.207170    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:16.380854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:16.380830    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:16.393592 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:16.393565    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:16.455954 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:16.455920    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:16.527794 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:16.527763    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:48:18.623347 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:18.623313    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:23.023305 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:23.023271    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:24.240739 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:24.240684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:24.412091 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:24.412054    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:26.471715 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:48:26.471679    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:48:27.425783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:27.425749    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:31.519211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.519177    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:48:31.522814 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.522790    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.541736 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.541708    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hlw\" (UniqueName: \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.541883 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.541818    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.541992 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.541977    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.542064 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.542010    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.571967 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.571926    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-aa20ad-serving-cert\""
Nov 03 12:48:31.576785 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.576760    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:48:31.643253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.643220    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.643568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.643543    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.643736 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:48:31.643660    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-serving-cert: secret "maas-api-e2e-extmod-b-aa20ad-serving-cert" not found
Nov 03 12:48:31.643841 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:48:31.643825    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls podName:3de16c0a-b248-4b6f-8072-27be9c0388d4 nodeName:}" failed. No retries permitted until 2026-11-03 12:48:32.143801969 +0000 UTC m=+3418.586004265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls") pod "maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" (UID: "3de16c0a-b248-4b6f-8072-27be9c0388d4") : secret "maas-api-e2e-extmod-b-aa20ad-serving-cert" not found
Nov 03 12:48:31.644084 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.644050    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.644167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.644147    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hlw\" (UniqueName: \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.644238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.644220    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.646806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.646769    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.696589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.696558    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hlw\" (UniqueName: \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:31.715585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.715552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:31.872854 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:31.872750    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:32.026282 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.026250    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:32.150557 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.150520    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:32.152961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.152935    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:32.438416 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.438317    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:32.808355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.808331    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:48:32.809846 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:48:32.809818    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de16c0a_b248_4b6f_8072_27be9c0388d4.slice/crio-6c816987f1ecb97f33328fe9372e9767cab8482bbe04c9d61d6067b81a521b23 WatchSource:0}: Error finding container 6c816987f1ecb97f33328fe9372e9767cab8482bbe04c9d61d6067b81a521b23: Status 404 returned error can't find the container with id 6c816987f1ecb97f33328fe9372e9767cab8482bbe04c9d61d6067b81a521b23
Nov 03 12:48:32.811211 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:32.811188    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:48:33.642115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:33.642077    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" event={"ID":"3de16c0a-b248-4b6f-8072-27be9c0388d4","Type":"ContainerStarted","Data":"6c816987f1ecb97f33328fe9372e9767cab8482bbe04c9d61d6067b81a521b23"}
Nov 03 12:48:34.652428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:34.652390    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" event={"ID":"3de16c0a-b248-4b6f-8072-27be9c0388d4","Type":"ContainerStarted","Data":"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"}
Nov 03 12:48:34.652850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:34.652582    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:34.761889 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:34.761826    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" podStartSLOduration=2.829700489 podStartE2EDuration="3.761808031s" totalImagesPullingTime="932.107542ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:48:34.759857687 +0000 UTC m=+3421.202059995" watchObservedRunningTime="2026-11-03 12:48:34.761808031 +0000 UTC m=+3421.204010337"
Nov 03 12:48:36.269834 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:36.269800    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:40.668360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:40.668328    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:48:40.731225 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:40.731194    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:45.093299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:45.093264    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:49.537504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:49.537450    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:51.503032 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:51.502987    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:52.690714 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:52.690681    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:52.855284 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:52.855248    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:53.010076 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.009993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:53.252747 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.252703    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:53.438218 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.438167    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:53.610112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.610057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:53.949614 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.949585    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:48:53.954938 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:53.954918    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.004670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.004641    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c84ae1-serving-cert\""
Nov 03 12:48:54.026755 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.026722    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:48:54.103543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.103508    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.103710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.103547    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtgrd\" (UniqueName: \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.103710 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.103676    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.103806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.103720    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.178740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.178708    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:54.204779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.204677    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.204779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.204728    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtgrd\" (UniqueName: \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.205002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.204807    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.205002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.204846    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.205722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.205688    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.207333 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.207304    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.207524 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.207507    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.262058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.262021    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtgrd\" (UniqueName: \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\") pod \"maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.269729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.269695    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:54.336165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.334551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:54.491144 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.491068    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:54.669418 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.669361    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:48:54.674509 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:48:54.674476    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a93097_1af2_4abb_adca_ea529c2c6609.slice/crio-86f1eb0bcd55f3b92331f8abc6b8c335cdc24c85fb0177f59eb7d44ea1effcdc WatchSource:0}: Error finding container 86f1eb0bcd55f3b92331f8abc6b8c335cdc24c85fb0177f59eb7d44ea1effcdc: Status 404 returned error can't find the container with id 86f1eb0bcd55f3b92331f8abc6b8c335cdc24c85fb0177f59eb7d44ea1effcdc
Nov 03 12:48:54.794846 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:54.794757    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" event={"ID":"93a93097-1af2-4abb-adca-ea529c2c6609","Type":"ContainerStarted","Data":"86f1eb0bcd55f3b92331f8abc6b8c335cdc24c85fb0177f59eb7d44ea1effcdc"}
Nov 03 12:48:56.473100 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:48:56.473021    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:48:56.809807 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:56.809721    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" event={"ID":"93a93097-1af2-4abb-adca-ea529c2c6609","Type":"ContainerStarted","Data":"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"}
Nov 03 12:48:56.809987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:56.809826    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:48:56.920408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:56.918198    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" podStartSLOduration=2.433564984 podStartE2EDuration="3.918181563s" totalImagesPullingTime="1.484616579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:48:56.914878801 +0000 UTC m=+3443.357081106" watchObservedRunningTime="2026-11-03 12:48:56.918181563 +0000 UTC m=+3443.360383868"
Nov 03 12:48:56.988851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:56.988801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:58.817185 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:58.817152    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:48:59.334139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:48:59.334106    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:02.823346 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:02.823315    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:49:03.218583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:03.218542    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:04.269990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:04.269944    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:07.605112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:07.605067    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:11.714431 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.714397    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:49:11.718362 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.718339    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.768566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.768528    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-38f178-serving-cert\""
Nov 03 12:49:11.810172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.810138    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2g7b\" (UniqueName: \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.810366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.810339    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.810519 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.810492    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.810586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.810570    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.817973 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.817951    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:49:11.911480 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.911437    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2g7b\" (UniqueName: \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.911687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.911497    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.911687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.911543    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.911687 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.911580    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.911861 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:49:11.911772    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-serving-cert: secret "maas-api-e2e-extmod-a-38f178-serving-cert" not found
Nov 03 12:49:11.911975 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:49:11.911863    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls podName:2fc5b740-3321-40af-8534-b5019aee5ced nodeName:}" failed. No retries permitted until 2026-11-03 12:49:12.411840456 +0000 UTC m=+3458.854042742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls") pod "maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" (UID: "2fc5b740-3321-40af-8534-b5019aee5ced") : secret "maas-api-e2e-extmod-a-38f178-serving-cert" not found
Nov 03 12:49:11.912271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.912249    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.913896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.913873    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:11.920158 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.920121    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:11.966982 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:11.966913    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2g7b\" (UniqueName: \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:12.073733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:12.073698    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:12.232540 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:12.232458    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:12.416820 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:12.416780    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:12.419290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:12.419257    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") pod \"maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:12.635096 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:12.635056    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:13.005112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:13.005085    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:49:13.005971 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:49:13.005944    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc5b740_3321_40af_8534_b5019aee5ced.slice/crio-dc9c28bee37cf82fc7b569df8c96e177fff21ff942825879fa77e281f1ad9369 WatchSource:0}: Error finding container dc9c28bee37cf82fc7b569df8c96e177fff21ff942825879fa77e281f1ad9369: Status 404 returned error can't find the container with id dc9c28bee37cf82fc7b569df8c96e177fff21ff942825879fa77e281f1ad9369
Nov 03 12:49:13.941174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:13.941138    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" event={"ID":"2fc5b740-3321-40af-8534-b5019aee5ced","Type":"ContainerStarted","Data":"dc9c28bee37cf82fc7b569df8c96e177fff21ff942825879fa77e281f1ad9369"}
Nov 03 12:49:14.950329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:14.950289    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" event={"ID":"2fc5b740-3321-40af-8534-b5019aee5ced","Type":"ContainerStarted","Data":"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"}
Nov 03 12:49:14.950758 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:14.950396    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:15.060421 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:15.060345    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" podStartSLOduration=3.198368556 podStartE2EDuration="4.060328855s" totalImagesPullingTime="861.960299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:49:15.056893383 +0000 UTC m=+3461.499095710" watchObservedRunningTime="2026-11-03 12:49:15.060328855 +0000 UTC m=+3461.502531162"
Nov 03 12:49:16.501169 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:16.501134    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:20.929396 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:20.929348    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:20.964857 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:20.964825    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:49:25.412850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:25.412801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:26.474563 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:49:26.474521    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:49:29.858131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:29.858019    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:34.209386 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:34.209340    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:38.591296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:38.591262    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:43.034654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:43.034592    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:47.538088 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:47.538051    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:51.879651 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:51.879618    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:56.301247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:56.301219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:49:56.473426 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:49:56.473387    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:49:59.652949 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:49:59.652916    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:00.801730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:00.801697    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:05.176541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:05.176501    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:08.226861 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:08.226820    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:09.551576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:09.551533    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:14.045310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:14.045212    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:14.245828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:14.245799    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:15.017905 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:15.017872    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:15.178776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:15.178734    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:17.182001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:17.181950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:17.336488 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:17.336452    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:19.140241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:19.140201    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:19.585153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:19.585066    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:23.712066 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:23.712032    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:26.473141 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:50:26.473098    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:50:27.118400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:27.118356    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:28.060916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:28.060883    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:28.968936 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:28.968890    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:29.233422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:29.233305    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:29.424460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:29.424420    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:30.341767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:30.341727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:30.494927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:30.494890    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:32.323953 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:32.323909    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:32.481961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:32.481926    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:32.644238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:32.644210    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:32.945108 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:32.945036    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:33.628115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:33.628083    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:33.784105 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:33.784076    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:37.545551 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:37.545516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:38.500816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:38.500782    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:41.895765 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:41.895730    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:42.051131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:42.051098    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:46.386885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:46.386855    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:47.485316 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:47.485278    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:47.643226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:47.643185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:47.994849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:47.994808    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:50.743771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:50.743736    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:53.518585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:53.518551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:55.045840 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:55.045800    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:56.473475 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:50:56.473444    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:50:59.420190 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:59.420151    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:50:59.965912 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:50:59.965876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:03.827779 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:03.827730    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:08.207182 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:08.207142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:12.575328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:12.575295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:16.940450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:16.940412    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:21.343833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:21.343803    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:25.047504 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.047469    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:51:25.047988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.047747    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" podUID="23a38c0b-dcae-46d7-9845-fb655db2fba0" containerName="maas-api" containerID="cri-o://fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648" gracePeriod=30
Nov 03 12:51:25.174525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.174476    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:25.323617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.323590    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:51:25.375300 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.375262    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\" (UniqueName: \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\") pod \"23a38c0b-dcae-46d7-9845-fb655db2fba0\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") "
Nov 03 12:51:25.375520 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.375455    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\") pod \"23a38c0b-dcae-46d7-9845-fb655db2fba0\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") "
Nov 03 12:51:25.375601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.375585    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") pod \"23a38c0b-dcae-46d7-9845-fb655db2fba0\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") "
Nov 03 12:51:25.375688 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.375660    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\") pod \"23a38c0b-dcae-46d7-9845-fb655db2fba0\" (UID: \"23a38c0b-dcae-46d7-9845-fb655db2fba0\") "
Nov 03 12:51:25.375819 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.375793    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca" pod "23a38c0b-dcae-46d7-9845-fb655db2fba0" (UID: "23a38c0b-dcae-46d7-9845-fb655db2fba0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:25.376085 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.376062    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23a38c0b-dcae-46d7-9845-fb655db2fba0-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:25.377570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.377548    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck" pod "23a38c0b-dcae-46d7-9845-fb655db2fba0" (UID: "23a38c0b-dcae-46d7-9845-fb655db2fba0"). InnerVolumeSpecName "kube-api-access-p2kck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:25.377764 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.377741    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs" pod "23a38c0b-dcae-46d7-9845-fb655db2fba0" (UID: "23a38c0b-dcae-46d7-9845-fb655db2fba0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:51:25.377989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.377974    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls" pod "23a38c0b-dcae-46d7-9845-fb655db2fba0" (UID: "23a38c0b-dcae-46d7-9845-fb655db2fba0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:51:25.477746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.477710    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:25.477746 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.477742    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23a38c0b-dcae-46d7-9845-fb655db2fba0-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:25.477964 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.477755    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2kck\" (UniqueName: \"kubernetes.io/projected/23a38c0b-dcae-46d7-9845-fb655db2fba0-kube-api-access-p2kck\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:25.932619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.932581    5784 generic.go:356] "Generic (PLEG): container finished" podID="23a38c0b-dcae-46d7-9845-fb655db2fba0" containerID="fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648" exitCode=0
Nov 03 12:51:25.932829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.932652    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"
Nov 03 12:51:25.932829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.932654    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" event={"ID":"23a38c0b-dcae-46d7-9845-fb655db2fba0","Type":"ContainerDied","Data":"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"}
Nov 03 12:51:25.932829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.932780    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz" event={"ID":"23a38c0b-dcae-46d7-9845-fb655db2fba0","Type":"ContainerDied","Data":"bed5b8359e28177558b034774e10a893e30161b8ef65c328b3006a263702cfb8"}
Nov 03 12:51:25.932829 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.932809    5784 scope.go:122] "RemoveContainer" containerID="fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"
Nov 03 12:51:25.948813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.948789    5784 scope.go:122] "RemoveContainer" containerID="fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"
Nov 03 12:51:25.949356 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:51:25.949325    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648\": container with ID starting with fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648 not found: ID does not exist" containerID="fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"
Nov 03 12:51:25.949549 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:25.949363    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648"} err="failed to get container status \"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648\": rpc error: code = NotFound desc = could not find container \"fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648\": container with ID starting with fee428a171af434530bf22e088a99a4090a41e146e29b948af2cbf694f09d648 not found: ID does not exist"
Nov 03 12:51:26.087329 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:26.087294    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:51:26.092109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:26.092081    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bebc6d-5d65d5dc46-4jqpz"]
Nov 03 12:51:26.473257 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:51:26.473216    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:51:26.998860 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:26.998825    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:27.682131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:27.682099    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a38c0b-dcae-46d7-9845-fb655db2fba0" path="/var/lib/kubelet/pods/23a38c0b-dcae-46d7-9845-fb655db2fba0/volumes"
Nov 03 12:51:31.408167 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:31.408138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:33.951900 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:33.951868    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:34.109601 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:34.109569    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:34.336597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:34.336497    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:51:34.356474 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:34.356444    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:51:35.391713 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:35.391683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:35.748956 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:35.748785    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:35.915868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:35.915828    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:36.073110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:36.073033    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:36.782503 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:36.782462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:40.723700 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:40.723619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:42.718902 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:42.718848    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:43.832109 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:43.832072    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:43.974465 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:43.974418    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:44.805126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:44.805091    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:51:44.805408 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:44.805329    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" podUID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" containerName="maas-api" containerID="cri-o://f49cd1f187dba488525a050fe5355772ff36e53bc42d7bbe2e45980700e9e96b" gracePeriod=30
Nov 03 12:51:44.923153 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:44.923121    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:45.084046 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.084010    5784 generic.go:356] "Generic (PLEG): container finished" podID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" containerID="f49cd1f187dba488525a050fe5355772ff36e53bc42d7bbe2e45980700e9e96b" exitCode=0
Nov 03 12:51:45.084293 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.084271    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" event={"ID":"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f","Type":"ContainerDied","Data":"f49cd1f187dba488525a050fe5355772ff36e53bc42d7bbe2e45980700e9e96b"}
Nov 03 12:51:45.114306 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.114274    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:51:45.308704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.308651    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\") pod \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") "
Nov 03 12:51:45.308704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.308713    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") pod \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") "
Nov 03 12:51:45.308946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.308817    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\" (UniqueName: \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\") pod \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") "
Nov 03 12:51:45.308946 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.308835    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\") pod \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\" (UID: \"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f\") "
Nov 03 12:51:45.309364 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.309328    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca" pod "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" (UID: "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:51:45.311493 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.311466    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs" pod "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" (UID: "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:51:45.311597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.311497    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs" pod "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" (UID: "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f"). InnerVolumeSpecName "kube-api-access-9rpfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:51:45.311597 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.311497    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls" pod "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" (UID: "6c18e3c2-fc1c-4a0a-9157-63d95cdc857f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:51:45.410273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.410220    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rpfs\" (UniqueName: \"kubernetes.io/projected/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-kube-api-access-9rpfs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:45.410273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.410269    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:45.410273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.410279    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:45.410273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:45.410289    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:51:46.092164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.092074    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw" event={"ID":"6c18e3c2-fc1c-4a0a-9157-63d95cdc857f","Type":"ContainerDied","Data":"cab8027bdbe5fc51c87855ea02d43d9b7882de23698fb96b5aa9ed4cdefd682c"}
Nov 03 12:51:46.092164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.092122    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"
Nov 03 12:51:46.092164 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.092146    5784 scope.go:122] "RemoveContainer" containerID="f49cd1f187dba488525a050fe5355772ff36e53bc42d7bbe2e45980700e9e96b"
Nov 03 12:51:46.245358 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.245323    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:51:46.291489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.291453    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fd7881-7668fcdf44-ztnfw"]
Nov 03 12:51:46.708035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:46.707998    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:47.682132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:47.682097    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" path="/var/lib/kubelet/pods/6c18e3c2-fc1c-4a0a-9157-63d95cdc857f/volumes"
Nov 03 12:51:48.213986 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:48.213950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:51.086976 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:51.086936    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:52.436813 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:52.436777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:52.592564 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:52.592530    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:53.868788 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:53.868753    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:54.337906 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:54.337874    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:55.078449 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:55.078409    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:55.799828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:51:55.799797    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:51:56.472794 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:51:56.472757    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:52:00.205740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:00.205683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:00.357426 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:00.357366    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:01.112002 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:01.111969    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:01.269436 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:01.269371    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:01.414852 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:01.414821    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:02.250694 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:02.250658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:02.406790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:02.406759    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:04.908319 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:04.908282    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:06.944216 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:06.944171    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:09.286619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:09.286584    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:13.849125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:13.849080    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:18.229631 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:18.229598    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:22.627016 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:22.626979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:26.472853 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:26.472813    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:52:27.029125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:27.029093    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:31.405147 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:31.405109    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:35.816326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:35.816288    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:40.234115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:40.234075    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:44.650026 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:44.649990    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:48.616264 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.616225    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-894e1d"]
Nov 03 12:52:48.616987 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.616968    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" containerName="maas-api"
Nov 03 12:52:48.617038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.616989    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" containerName="maas-api"
Nov 03 12:52:48.617038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.617032    5784 cpu_manager.go:414] "containerMap: removing container" podUID="23a38c0b-dcae-46d7-9845-fb655db2fba0" containerName="maas-api"
Nov 03 12:52:48.617106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.617038    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a38c0b-dcae-46d7-9845-fb655db2fba0" containerName="maas-api"
Nov 03 12:52:48.617140 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.617126    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c18e3c2-fc1c-4a0a-9157-63d95cdc857f" containerName="maas-api"
Nov 03 12:52:48.617173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.617143    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a38c0b-dcae-46d7-9845-fb655db2fba0" containerName="maas-api"
Nov 03 12:52:48.621817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.621802    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:48.668877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.668838    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-894e1d"]
Nov 03 12:52:48.711157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.711127    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 12:52:48.746405 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.746320    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpm2\" (UniqueName: \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\") pod \"test-curl-7593-894e1d\" (UID: \"8cff4a8f-86c3-4756-b71c-39b3bcfe9992\") " pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:48.847927 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.847887    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpm2\" (UniqueName: \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\") pod \"test-curl-7593-894e1d\" (UID: \"8cff4a8f-86c3-4756-b71c-39b3bcfe9992\") " pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:48.900365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.900330    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpm2\" (UniqueName: \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\") pod \"test-curl-7593-894e1d\" (UID: \"8cff4a8f-86c3-4756-b71c-39b3bcfe9992\") " pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:48.933708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:48.933679    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:49.017729 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:49.017700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:49.297650 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:49.297621    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-894e1d"]
Nov 03 12:52:49.298729 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:52:49.298698    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cff4a8f_86c3_4756_b71c_39b3bcfe9992.slice/crio-2020eb75aa782700935bb58f2b26fbef531d311c68eb8d037c04a15e6194301b WatchSource:0}: Error finding container 2020eb75aa782700935bb58f2b26fbef531d311c68eb8d037c04a15e6194301b: Status 404 returned error can't find the container with id 2020eb75aa782700935bb58f2b26fbef531d311c68eb8d037c04a15e6194301b
Nov 03 12:52:49.565489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:49.565372    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-894e1d" event={"ID":"8cff4a8f-86c3-4756-b71c-39b3bcfe9992","Type":"ContainerStarted","Data":"d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5"}
Nov 03 12:52:49.565489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:49.565449    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-894e1d" event={"ID":"8cff4a8f-86c3-4756-b71c-39b3bcfe9992","Type":"ContainerStarted","Data":"2020eb75aa782700935bb58f2b26fbef531d311c68eb8d037c04a15e6194301b"}
Nov 03 12:52:49.671623 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:49.671555    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-894e1d" podStartSLOduration=1.6715380309999999 podStartE2EDuration="1.671538031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:52:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:52:49.66936266 +0000 UTC m=+3676.111564989" watchObservedRunningTime="2026-11-03 12:52:49.671538031 +0000 UTC m=+3676.113740338"
Nov 03 12:52:51.067275 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:51.067234    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-894e1d"]
Nov 03 12:52:51.071797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:51.071760    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-894e1d"]
Nov 03 12:52:51.579305 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:51.579269    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-894e1d" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" containerName="test-curl-7593-894e1d" containerID="cri-o://d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5" gracePeriod=2
Nov 03 12:52:51.626424 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:51.626371    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-894e1d\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:53.428198 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:53.428113    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:53.812175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:53.812149    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:53.859427 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:53.859359    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-894e1d\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:53.905004 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:53.904967    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\" (UniqueName: \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\") pod \"8cff4a8f-86c3-4756-b71c-39b3bcfe9992\" (UID: \"8cff4a8f-86c3-4756-b71c-39b3bcfe9992\") "
Nov 03 12:52:53.907123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:53.907095    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2" pod "8cff4a8f-86c3-4756-b71c-39b3bcfe9992" (UID: "8cff4a8f-86c3-4756-b71c-39b3bcfe9992"). InnerVolumeSpecName "kube-api-access-vrpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:54.006556 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.006466    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrpm2\" (UniqueName: \"kubernetes.io/projected/8cff4a8f-86c3-4756-b71c-39b3bcfe9992-kube-api-access-vrpm2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:54.603062 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.603024    5784 generic.go:356] "Generic (PLEG): container finished" podID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" containerID="d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5" exitCode=137
Nov 03 12:52:54.603538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.603067    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:54.603538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.603120    5784 scope.go:122] "RemoveContainer" containerID="d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5"
Nov 03 12:52:54.613841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.613821    5784 scope.go:122] "RemoveContainer" containerID="d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5"
Nov 03 12:52:54.614140 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:54.614114    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5\": container with ID starting with d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5 not found: ID does not exist" containerID="d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5"
Nov 03 12:52:54.614210 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:54.614153    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5"} err="failed to get container status \"d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5\": rpc error: code = NotFound desc = could not find container \"d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5\": container with ID starting with d47fed1db30dd5d337979323669034cd6d514cc52fe73a0495432a659d4c63b5 not found: ID does not exist"
Nov 03 12:52:54.650589 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:54.650543    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-894e1d\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:54.697994 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:54.697952    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-894e1d\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" pod="opendatahub/test-curl-7593-894e1d"
Nov 03 12:52:55.682150 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:55.682120    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" path="/var/lib/kubelet/pods/8cff4a8f-86c3-4756-b71c-39b3bcfe9992/volumes"
Nov 03 12:52:56.314363 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:56.314325    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:56.475561 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:56.475526    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:52:57.136789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.136751    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:52:57.137226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.137013    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" podUID="3de16c0a-b248-4b6f-8072-27be9c0388d4" containerName="maas-api" containerID="cri-o://3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773" gracePeriod=30
Nov 03 12:52:57.260833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.260779    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:57.437538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.437207    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:52:57.551734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.551700    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\" (UniqueName: \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\") pod \"3de16c0a-b248-4b6f-8072-27be9c0388d4\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") "
Nov 03 12:52:57.551937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.551762    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\") pod \"3de16c0a-b248-4b6f-8072-27be9c0388d4\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") "
Nov 03 12:52:57.551937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.551870    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") pod \"3de16c0a-b248-4b6f-8072-27be9c0388d4\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") "
Nov 03 12:52:57.552053 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.551992    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\") pod \"3de16c0a-b248-4b6f-8072-27be9c0388d4\" (UID: \"3de16c0a-b248-4b6f-8072-27be9c0388d4\") "
Nov 03 12:52:57.552822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.552792    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca" pod "3de16c0a-b248-4b6f-8072-27be9c0388d4" (UID: "3de16c0a-b248-4b6f-8072-27be9c0388d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:52:57.555489 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.555451    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls" pod "3de16c0a-b248-4b6f-8072-27be9c0388d4" (UID: "3de16c0a-b248-4b6f-8072-27be9c0388d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:52:57.558931 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.558896    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs" pod "3de16c0a-b248-4b6f-8072-27be9c0388d4" (UID: "3de16c0a-b248-4b6f-8072-27be9c0388d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:52:57.559877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.559843    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw" pod "3de16c0a-b248-4b6f-8072-27be9c0388d4" (UID: "3de16c0a-b248-4b6f-8072-27be9c0388d4"). InnerVolumeSpecName "kube-api-access-j9hlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:52:57.628988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.628953    5784 generic.go:356] "Generic (PLEG): container finished" podID="3de16c0a-b248-4b6f-8072-27be9c0388d4" containerID="3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773" exitCode=0
Nov 03 12:52:57.629166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.629005    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"
Nov 03 12:52:57.629166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.629044    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" event={"ID":"3de16c0a-b248-4b6f-8072-27be9c0388d4","Type":"ContainerDied","Data":"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"}
Nov 03 12:52:57.629166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.629110    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm" event={"ID":"3de16c0a-b248-4b6f-8072-27be9c0388d4","Type":"ContainerDied","Data":"6c816987f1ecb97f33328fe9372e9767cab8482bbe04c9d61d6067b81a521b23"}
Nov 03 12:52:57.629166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.629139    5784 scope.go:122] "RemoveContainer" containerID="3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"
Nov 03 12:52:57.641862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.641841    5784 scope.go:122] "RemoveContainer" containerID="3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"
Nov 03 12:52:57.642178 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:52:57.642155    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773\": container with ID starting with 3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773 not found: ID does not exist" containerID="3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"
Nov 03 12:52:57.642280 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.642185    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773"} err="failed to get container status \"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773\": rpc error: code = NotFound desc = could not find container \"3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773\": container with ID starting with 3787178de3c39a1ce0d95accd33f09e7d5e1f73fc027262892a6a6b66e7c1773 not found: ID does not exist"
Nov 03 12:52:57.653462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.653393    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3de16c0a-b248-4b6f-8072-27be9c0388d4-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:57.653462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.653419    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9hlw\" (UniqueName: \"kubernetes.io/projected/3de16c0a-b248-4b6f-8072-27be9c0388d4-kube-api-access-j9hlw\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:57.653462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.653432    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:57.653462 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.653441    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3de16c0a-b248-4b6f-8072-27be9c0388d4-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:52:57.782112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.782076    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:52:57.785253 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:57.785222    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-aa20ad-d67c648bd-gftxm"]
Nov 03 12:52:58.906966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:58.906930    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:52:59.681945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:52:59.681911    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de16c0a-b248-4b6f-8072-27be9c0388d4" path="/var/lib/kubelet/pods/3de16c0a-b248-4b6f-8072-27be9c0388d4/volumes"
Nov 03 12:53:00.628737 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:00.628692    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:02.407915 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:02.407879    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:02.566797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:02.566740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:03.427389 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:03.427353    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:03.761033 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:03.760943    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:04.248971 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:04.248938    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:06.216586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:06.216552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:06.367022 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:06.366979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:08.234355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:08.234314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:09.933961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:09.933928    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:12.126510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.126405    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:53:12.127072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.126665    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" podUID="93a93097-1af2-4abb-adca-ea529c2c6609" containerName="maas-api" containerID="cri-o://be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1" gracePeriod=30
Nov 03 12:53:12.240103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.240079    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:12.394166 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.394100    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:53:12.425676 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.425639    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\" (UniqueName: \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\") pod \"93a93097-1af2-4abb-adca-ea529c2c6609\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") "
Nov 03 12:53:12.425872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.425692    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\") pod \"93a93097-1af2-4abb-adca-ea529c2c6609\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") "
Nov 03 12:53:12.425872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.425760    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\") pod \"93a93097-1af2-4abb-adca-ea529c2c6609\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") "
Nov 03 12:53:12.425872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.425840    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\") pod \"93a93097-1af2-4abb-adca-ea529c2c6609\" (UID: \"93a93097-1af2-4abb-adca-ea529c2c6609\") "
Nov 03 12:53:12.426140 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.426108    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca" pod "93a93097-1af2-4abb-adca-ea529c2c6609" (UID: "93a93097-1af2-4abb-adca-ea529c2c6609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:12.427767 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.427740    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd" pod "93a93097-1af2-4abb-adca-ea529c2c6609" (UID: "93a93097-1af2-4abb-adca-ea529c2c6609"). InnerVolumeSpecName "kube-api-access-vtgrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:12.427863 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.427836    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls" pod "93a93097-1af2-4abb-adca-ea529c2c6609" (UID: "93a93097-1af2-4abb-adca-ea529c2c6609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:12.427951 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.427931    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs" pod "93a93097-1af2-4abb-adca-ea529c2c6609" (UID: "93a93097-1af2-4abb-adca-ea529c2c6609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:12.526888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.526850    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:12.526888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.526888    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/93a93097-1af2-4abb-adca-ea529c2c6609-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:12.527111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.526904    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtgrd\" (UniqueName: \"kubernetes.io/projected/93a93097-1af2-4abb-adca-ea529c2c6609-kube-api-access-vtgrd\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:12.527111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.526919    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93a93097-1af2-4abb-adca-ea529c2c6609-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:12.742476 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.742365    5784 generic.go:356] "Generic (PLEG): container finished" podID="93a93097-1af2-4abb-adca-ea529c2c6609" containerID="be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1" exitCode=0
Nov 03 12:53:12.742476 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.742449    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"
Nov 03 12:53:12.742693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.742449    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" event={"ID":"93a93097-1af2-4abb-adca-ea529c2c6609","Type":"ContainerDied","Data":"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"}
Nov 03 12:53:12.742693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.742571    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2" event={"ID":"93a93097-1af2-4abb-adca-ea529c2c6609","Type":"ContainerDied","Data":"86f1eb0bcd55f3b92331f8abc6b8c335cdc24c85fb0177f59eb7d44ea1effcdc"}
Nov 03 12:53:12.742693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.742598    5784 scope.go:122] "RemoveContainer" containerID="be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"
Nov 03 12:53:12.754914 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.754877    5784 scope.go:122] "RemoveContainer" containerID="be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"
Nov 03 12:53:12.755270 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:12.755244    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1\": container with ID starting with be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1 not found: ID does not exist" containerID="be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"
Nov 03 12:53:12.755366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.755282    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1"} err="failed to get container status \"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1\": rpc error: code = NotFound desc = could not find container \"be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1\": container with ID starting with be5708397c9842980fae0379e3f71264e9d7085c416e176e66f53c245d1194f1 not found: ID does not exist"
Nov 03 12:53:12.897587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.897521    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:53:12.942640 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:12.942605    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c84ae1-74b9bf9999-nm8p2"]
Nov 03 12:53:13.683069 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:13.683034    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a93097-1af2-4abb-adca-ea529c2c6609" path="/var/lib/kubelet/pods/93a93097-1af2-4abb-adca-ea529c2c6609/volumes"
Nov 03 12:53:13.824014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:13.823979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:18.326698 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:18.326661    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:21.004060 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:21.004027    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:21.157393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:21.157348    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:21.945256 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:21.945218    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:53:21.990734 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:21.990705    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:22.413247 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:22.413215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:22.924812 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:22.924777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:23.109799 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:23.109756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:24.402996 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:24.402953    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:24.583975 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:24.583942    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:26.472238 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:26.472199    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:53:27.006542 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.006505    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:53:27.006759 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.006714    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" podUID="2fc5b740-3321-40af-8534-b5019aee5ced" containerName="maas-api" containerID="cri-o://f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981" gracePeriod=30
Nov 03 12:53:27.126641 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.126533    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:27.289872 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.289844    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:53:27.420528 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.420490    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\") pod \"2fc5b740-3321-40af-8534-b5019aee5ced\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") "
Nov 03 12:53:27.420727 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.420639    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\") pod \"2fc5b740-3321-40af-8534-b5019aee5ced\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") "
Nov 03 12:53:27.420970 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.420785    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\" (UniqueName: \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\") pod \"2fc5b740-3321-40af-8534-b5019aee5ced\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") "
Nov 03 12:53:27.420970 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.420813    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") pod \"2fc5b740-3321-40af-8534-b5019aee5ced\" (UID: \"2fc5b740-3321-40af-8534-b5019aee5ced\") "
Nov 03 12:53:27.422282 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.422242    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca" pod "2fc5b740-3321-40af-8534-b5019aee5ced" (UID: "2fc5b740-3321-40af-8534-b5019aee5ced"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:53:27.426038 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.426000    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs" pod "2fc5b740-3321-40af-8534-b5019aee5ced" (UID: "2fc5b740-3321-40af-8534-b5019aee5ced"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:27.426273 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.426254    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b" pod "2fc5b740-3321-40af-8534-b5019aee5ced" (UID: "2fc5b740-3321-40af-8534-b5019aee5ced"). InnerVolumeSpecName "kube-api-access-x2g7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:27.439584 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.438487    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls" pod "2fc5b740-3321-40af-8534-b5019aee5ced" (UID: "2fc5b740-3321-40af-8534-b5019aee5ced"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:27.522642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.522553    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2g7b\" (UniqueName: \"kubernetes.io/projected/2fc5b740-3321-40af-8534-b5019aee5ced-kube-api-access-x2g7b\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:27.522642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.522590    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:27.522642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.522607    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc5b740-3321-40af-8534-b5019aee5ced-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:27.522642 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.522620    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc5b740-3321-40af-8534-b5019aee5ced-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:27.867023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.866934    5784 generic.go:356] "Generic (PLEG): container finished" podID="2fc5b740-3321-40af-8534-b5019aee5ced" containerID="f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981" exitCode=0
Nov 03 12:53:27.867023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.866984    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"
Nov 03 12:53:27.867023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.867002    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" event={"ID":"2fc5b740-3321-40af-8534-b5019aee5ced","Type":"ContainerDied","Data":"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"}
Nov 03 12:53:27.867294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.867061    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6" event={"ID":"2fc5b740-3321-40af-8534-b5019aee5ced","Type":"ContainerDied","Data":"dc9c28bee37cf82fc7b569df8c96e177fff21ff942825879fa77e281f1ad9369"}
Nov 03 12:53:27.867294 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.867086    5784 scope.go:122] "RemoveContainer" containerID="f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"
Nov 03 12:53:27.879474 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.879454    5784 scope.go:122] "RemoveContainer" containerID="f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"
Nov 03 12:53:27.879760 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:27.879740    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981\": container with ID starting with f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981 not found: ID does not exist" containerID="f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"
Nov 03 12:53:27.879853 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:27.879770    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981"} err="failed to get container status \"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981\": rpc error: code = NotFound desc = could not find container \"f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981\": container with ID starting with f34a48a2227df67d4a1135ec7dcc3c70e808fbb1968d80efa1f21037b2511981 not found: ID does not exist"
Nov 03 12:53:28.020147 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:28.020109    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:53:28.071537 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:28.071499    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-38f178-54c7bd87bc-f98z6"]
Nov 03 12:53:28.686199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:28.686156    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:28.852637 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:28.852604    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:29.156016 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:29.155979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:29.697790 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:29.697753    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc5b740-3321-40af-8534-b5019aee5ced" path="/var/lib/kubelet/pods/2fc5b740-3321-40af-8534-b5019aee5ced/volumes"
Nov 03 12:53:32.246795 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:32.246756    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:32.403862 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:32.403822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:33.628784 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:33.628750    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:33.870221 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:33.870166    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:34.017797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:34.017697    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:36.968081 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:36.968032    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" podUID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" containerName="main" containerID="cri-o://2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a" gracePeriod=45
Nov 03 12:53:37.227943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.227874    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:53:37.335748 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.335711    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.335943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.335813    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.335943 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.335884    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\" (UniqueName: \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.336047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.335945    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.336047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.336003    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.336047 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.336034    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\") pod \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\" (UID: \"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a\") "
Nov 03 12:53:37.337725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.337683    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:37.337893 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.337761    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:37.337974 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.337921    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:37.339134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.339105    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "kube-api-access-6jj7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:37.339868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.339837    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:53:37.340018 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.339991    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs" pod "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" (UID: "3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 12:53:37.437960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437926    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.437960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437954    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.437960 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437966    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jj7c\" (UniqueName: \"kubernetes.io/projected/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-kube-api-access-6jj7c\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.438200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437975    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.438200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437984    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.438200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.437994    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:37.951443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.951340    5784 generic.go:356] "Generic (PLEG): container finished" podID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" containerID="2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a" exitCode=0
Nov 03 12:53:37.951443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.951417    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"
Nov 03 12:53:37.951443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.951422    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" event={"ID":"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a","Type":"ContainerDied","Data":"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"}
Nov 03 12:53:37.951696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.951483    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz" event={"ID":"3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a","Type":"ContainerDied","Data":"4f5623993984274c253e68b1177172b71cecb4d54e3a54894fe41f4dd9762b03"}
Nov 03 12:53:37.951696 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.951511    5784 scope.go:122] "RemoveContainer" containerID="2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"
Nov 03 12:53:37.964580 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.964551    5784 scope.go:122] "RemoveContainer" containerID="2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"
Nov 03 12:53:37.964847 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:37.964830    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a\": container with ID starting with 2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a not found: ID does not exist" containerID="2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"
Nov 03 12:53:37.964911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:37.964855    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a"} err="failed to get container status \"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a\": rpc error: code = NotFound desc = could not find container \"2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a\": container with ID starting with 2729b5d262db2ca8f853d0a0509591b808de7d5385d6543055c63b12ef22355a not found: ID does not exist"
Nov 03 12:53:38.103822 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:38.103783    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:53:38.151428 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:38.151390    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-a4489c/auth-test-model-a4489c-kserve-7bcb4bd89c-4zsxz"]
Nov 03 12:53:38.233014 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:38.232928    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:39.682059 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:39.682020    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" path="/var/lib/kubelet/pods/3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a/volumes"
Nov 03 12:53:42.722681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:42.722650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:47.086495 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:47.086436    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:51.490461 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:51.490426    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:52.541139 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541105    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7593-bd9b49"]
Nov 03 12:53:52.541670 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541657    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" containerName="main"
Nov 03 12:53:52.541715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541671    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" containerName="main"
Nov 03 12:53:52.541715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541693    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3de16c0a-b248-4b6f-8072-27be9c0388d4" containerName="maas-api"
Nov 03 12:53:52.541715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541698    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de16c0a-b248-4b6f-8072-27be9c0388d4" containerName="maas-api"
Nov 03 12:53:52.541715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541706    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" containerName="test-curl-7593-894e1d"
Nov 03 12:53:52.541715 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541712    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" containerName="test-curl-7593-894e1d"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541723    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2fc5b740-3321-40af-8534-b5019aee5ced" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541727    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc5b740-3321-40af-8534-b5019aee5ced" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541747    5784 cpu_manager.go:414] "containerMap: removing container" podUID="93a93097-1af2-4abb-adca-ea529c2c6609" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541752    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a93097-1af2-4abb-adca-ea529c2c6609" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541828    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc8e96f-6e6a-46ef-b6de-2e1b28732e2a" containerName="main"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541840    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de16c0a-b248-4b6f-8072-27be9c0388d4" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541846    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cff4a8f-86c3-4756-b71c-39b3bcfe9992" containerName="test-curl-7593-894e1d"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541853    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc5b740-3321-40af-8534-b5019aee5ced" containerName="maas-api"
Nov 03 12:53:52.541873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.541860    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a93097-1af2-4abb-adca-ea529c2c6609" containerName="maas-api"
Nov 03 12:53:52.545200 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.545180    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:52.595299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.595267    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-bd9b49"]
Nov 03 12:53:52.637366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.637333    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 12:53:52.710816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.710781    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8q8\" (UniqueName: \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\") pod \"test-curl-7593-bd9b49\" (UID: \"11b912a6-33ec-4b3a-b6c6-98377907134b\") " pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:52.812618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.812528    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8q8\" (UniqueName: \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\") pod \"test-curl-7593-bd9b49\" (UID: \"11b912a6-33ec-4b3a-b6c6-98377907134b\") " pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:52.865086 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:52.865046    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8q8\" (UniqueName: \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\") pod \"test-curl-7593-bd9b49\" (UID: \"11b912a6-33ec-4b3a-b6c6-98377907134b\") " pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:53.157563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:53.157522    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:53.523591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:53.523566    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7593-bd9b49"]
Nov 03 12:53:53.524886 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:53:53.524863    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11b912a6_33ec_4b3a_b6c6_98377907134b.slice/crio-d9a3657b7a969b56e6bb8f584cd9b8c3a9ec99321fe084467e153781d481d740 WatchSource:0}: Error finding container d9a3657b7a969b56e6bb8f584cd9b8c3a9ec99321fe084467e153781d481d740: Status 404 returned error can't find the container with id d9a3657b7a969b56e6bb8f584cd9b8c3a9ec99321fe084467e153781d481d740
Nov 03 12:53:54.066916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:54.066880    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-bd9b49" event={"ID":"11b912a6-33ec-4b3a-b6c6-98377907134b","Type":"ContainerStarted","Data":"5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597"}
Nov 03 12:53:54.066916 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:54.066921    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7593-bd9b49" event={"ID":"11b912a6-33ec-4b3a-b6c6-98377907134b","Type":"ContainerStarted","Data":"d9a3657b7a969b56e6bb8f584cd9b8c3a9ec99321fe084467e153781d481d740"}
Nov 03 12:53:54.171964 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:54.171902    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7593-bd9b49" podStartSLOduration=2.171887419 podStartE2EDuration="2.171887419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:54.169364362 +0000 UTC m=+3740.611566698" watchObservedRunningTime="2026-11-03 12:53:54.171887419 +0000 UTC m=+3740.614089724"
Nov 03 12:53:55.119473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.119440    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 12:53:55.123805 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.123782    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.171968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.171938    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dg57t\""
Nov 03 12:53:55.243296 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243221    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243352    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243416    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhhpt\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243496 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243444    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243709 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243588    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243709 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243641    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243709 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243676    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243718    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.243869 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.243747    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.314594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.314560    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 12:53:55.345313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345275    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345397    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345444    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345479    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345514    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345543 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345536    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345575    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345841 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345825    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.345963 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.345856    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.346282 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.346255    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.346409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.346261    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.346409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.346334    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.346409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.346338    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhhpt\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.346594 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.346477    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.348275 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.348248    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.348989 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.348966    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.399201 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.399170    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.399691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.399668    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhhpt\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vw59m\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.438143 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.438115    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:55.633836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.633800    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7593-bd9b49"]
Nov 03 12:53:55.636870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.636831    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7593-bd9b49"]
Nov 03 12:53:55.857197 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.857168    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 12:53:55.857943 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:53:55.857913    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01468df6_1754_4cbe_b664_03e846b35b71.slice/crio-7adc98162831d92c157408f89b88f771bb830c812c543e75daccfe286873c1f1 WatchSource:0}: Error finding container 7adc98162831d92c157408f89b88f771bb830c812c543e75daccfe286873c1f1: Status 404 returned error can't find the container with id 7adc98162831d92c157408f89b88f771bb830c812c543e75daccfe286873c1f1
Nov 03 12:53:55.860868 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.860834    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:53:55.860945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.860906    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:53:55.860945 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.860937    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:53:55.875663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:55.875641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:56.083466 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.083426    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" event={"ID":"01468df6-1754-4cbe-b664-03e846b35b71","Type":"ContainerStarted","Data":"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"}
Nov 03 12:53:56.083680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.083491    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" event={"ID":"01468df6-1754-4cbe-b664-03e846b35b71","Type":"ContainerStarted","Data":"7adc98162831d92c157408f89b88f771bb830c812c543e75daccfe286873c1f1"}
Nov 03 12:53:56.083680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.083526    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7593-bd9b49" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" containerName="test-curl-7593-bd9b49" containerID="cri-o://5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597" gracePeriod=2
Nov 03 12:53:56.191624 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.191570    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podStartSLOduration=1.191555873 podStartE2EDuration="1.191555873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:53:56.190522712 +0000 UTC m=+3742.632725037" watchObservedRunningTime="2026-11-03 12:53:56.191555873 +0000 UTC m=+3742.633758178"
Nov 03 12:53:56.439021 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.438974    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:53:56.440298 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.440268    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:53:56.440475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:56.440327    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:53:56.488447 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:56.483708    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:53:57.058360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:57.058331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:57.221355 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:57.221318    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:53:57.438874 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:57.438833    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:53:57.439199 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:57.439169    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:53:58.367338 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.367305    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:58.374660 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.374624    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\" (UniqueName: \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\") pod \"11b912a6-33ec-4b3a-b6c6-98377907134b\" (UID: \"11b912a6-33ec-4b3a-b6c6-98377907134b\") "
Nov 03 12:53:58.376672 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.376650    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8" pod "11b912a6-33ec-4b3a-b6c6-98377907134b" (UID: "11b912a6-33ec-4b3a-b6c6-98377907134b"). InnerVolumeSpecName "kube-api-access-sq8q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:53:58.440924 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.440874    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:53:58.441151 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.440947    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:53:58.476238 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:58.476203    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8q8\" (UniqueName: \"kubernetes.io/projected/11b912a6-33ec-4b3a-b6c6-98377907134b-kube-api-access-sq8q8\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:53:59.108587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.108553    5784 generic.go:356] "Generic (PLEG): container finished" podID="11b912a6-33ec-4b3a-b6c6-98377907134b" containerID="5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597" exitCode=137
Nov 03 12:53:59.108811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.108605    5784 scope.go:122] "RemoveContainer" containerID="5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597"
Nov 03 12:53:59.108811 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.108631    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:59.146326 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.146304    5784 scope.go:122] "RemoveContainer" containerID="5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597"
Nov 03 12:53:59.146698 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:59.146679    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597\": container with ID starting with 5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597 not found: ID does not exist" containerID="5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597"
Nov 03 12:53:59.146797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.146705    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597"} err="failed to get container status \"5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597\": rpc error: code = NotFound desc = could not find container \"5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597\": container with ID starting with 5b4975263514ad42e86d4e27cad853d4ae50566c5c1a472b7dbb0af290ce0597 not found: ID does not exist"
Nov 03 12:53:59.193525 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:53:59.193479    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7593-bd9b49\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" pod="opendatahub/test-curl-7593-bd9b49"
Nov 03 12:53:59.440881 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.440841    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:53:59.441435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.440914    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:53:59.682134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:53:59.682096    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" path="/var/lib/kubelet/pods/11b912a6-33ec-4b3a-b6c6-98377907134b/volumes"
Nov 03 12:54:00.349050 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:00.348998    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:00.438802 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:00.438764    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:54:00.439075 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:00.439048    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:54:00.964056 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:00.964026    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:01.440788 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:01.440745    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:54:01.441175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:01.440826    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:54:02.438701 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:02.438658    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:54:02.439217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:02.438755    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:54:03.449217 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:03.449179    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vw59m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 12:54:03.449871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:03.449833    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 12:54:03.463270 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:03.463235    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:03.615339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:03.615304    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:04.580849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:04.580815    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:54:04.581820 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:04.581799    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:54:04.582850 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:04.582830    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 12:54:04.817208 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:04.817151    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:07.715241 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:07.715211    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:08.414633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:08.414597    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:08.775174 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:08.775085    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:09.336352 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:09.336311    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:11.602272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:11.602235    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:12.070722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:12.070674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:12.236517 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:12.236487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:12.390031 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:12.389992    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:14.113663 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:14.113631    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:15.123681 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:15.123651    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:15.285576 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:15.285546    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:18.704127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:18.704099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:23.138226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:23.138159    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:26.477413 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:54:26.477344    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:54:27.610899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:27.610861    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:32.071691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:32.071659    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:36.465339 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:36.465281    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:40.366730 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:40.366694    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:40.519789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:40.519752    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:40.899679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:40.899648    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:42.620619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:42.620575    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:43.271260 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:43.271215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:43.510207 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:43.510176    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:45.320157 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:45.320123    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:47.700420 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:47.700388    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:47.859450 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:47.859420    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:48.019830 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:48.019755    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:49.985596 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:49.985559    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:50.768965 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:50.768927    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:50.933437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:50.933400    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:54.509534 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:54.509500    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:55.558165 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.558070    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 12:54:55.558858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.558838    5784 cpu_manager.go:414] "containerMap: removing container" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" containerName="test-curl-7593-bd9b49"
Nov 03 12:54:55.558930 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.558859    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" containerName="test-curl-7593-bd9b49"
Nov 03 12:54:55.559003 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.558983    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b912a6-33ec-4b3a-b6c6-98377907134b" containerName="test-curl-7593-bd9b49"
Nov 03 12:54:55.562583 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.562559    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700173 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700139    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700190    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700216    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700316    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700353    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700403    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md9lr\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700544    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700589 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700582    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.700724 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.700663    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.755836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.755802    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 12:54:55.801680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801645    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.801866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801703    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.801866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801740    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.801866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801773    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.801866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801810    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.801866 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801836    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md9lr\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801874    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.801900    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802078    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802116    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802117    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802299    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802437 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802340    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.802668 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.802651    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.804231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.804211    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.804412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.804393    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.855299 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.855225    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md9lr\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.856998 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.856975    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-dspjl\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:55.876648 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:55.876612    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:56.294899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.294859    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 12:54:56.295740 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:54:56.295712    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2952de83_4b48_4b89_ba3d_4d2f9deb4e32.slice/crio-aa282e00d6b5ec18576f79c9932508af9bab170e3a475bb2e53dd915f51e7b7f WatchSource:0}: Error finding container aa282e00d6b5ec18576f79c9932508af9bab170e3a475bb2e53dd915f51e7b7f: Status 404 returned error can't find the container with id aa282e00d6b5ec18576f79c9932508af9bab170e3a475bb2e53dd915f51e7b7f
Nov 03 12:54:56.298215 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.298183    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:54:56.298304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.298245    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:54:56.298304 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.298276    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:54:56.475128 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:54:56.475093    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:54:56.568725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.568624    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" event={"ID":"2952de83-4b48-4b89-ba3d-4d2f9deb4e32","Type":"ContainerStarted","Data":"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"}
Nov 03 12:54:56.568725 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.568683    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" event={"ID":"2952de83-4b48-4b89-ba3d-4d2f9deb4e32","Type":"ContainerStarted","Data":"aa282e00d6b5ec18576f79c9932508af9bab170e3a475bb2e53dd915f51e7b7f"}
Nov 03 12:54:56.678409 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.678329    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podStartSLOduration=1.678314842 podStartE2EDuration="1.678314842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:54:56.676493833 +0000 UTC m=+3803.118696163" watchObservedRunningTime="2026-11-03 12:54:56.678314842 +0000 UTC m=+3803.120517146"
Nov 03 12:54:56.876851 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:56.876821    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:54:57.879394 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:57.879348    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:54:57.879884 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:57.879428    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:54:58.878094 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:58.878045    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:54:58.878290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:58.878123    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:54:59.041400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:59.040398    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:54:59.877350 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:59.877308    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:54:59.877546 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:54:59.877392    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:55:00.877976 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:00.877934    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:55:00.878630 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:00.878596    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:55:01.360619 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:01.360513    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:01.877473 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:01.877438    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:55:01.877655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:01.877489    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:55:02.877825 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:02.877790    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:55:02.878317 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:02.878289    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:55:03.518041 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:03.517993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:03.877240 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:03.877195    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 03 12:55:03.877445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:03.877265    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 03 12:55:05.878612 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:05.878567    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:05.879048 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:05.878669    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:06.928585 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:06.928541    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:06.929290 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:06.929249    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:07.705879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:07.705840    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:07.852111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:07.852051    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:07.970691 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:07.970573    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:55:07.971308 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:07.971272    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:55:08.018475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:08.018424    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:08.163653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:08.163578    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:55:08.164035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:08.164012    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:55:08.164804 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:08.164783    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 12:55:10.661020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.660976    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"]
Nov 03 12:55:10.666271 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.666246    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.720919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.720880    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721006    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721103 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721083    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721318 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721292    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721340    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthlb\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-kube-api-access-jthlb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721490 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721443    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec043c-9364-4fa0-a313-a236c85aa364-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721622 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721530    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.721925 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721898    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec043c-9364-4fa0-a313-a236c85aa364-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.722040 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.721948    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.766544 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.766499    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"]
Nov 03 12:55:10.823446 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823409    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.823656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823472    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.823656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823520    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.823656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823556    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthlb\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-kube-api-access-jthlb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.823656 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823609    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec043c-9364-4fa0-a313-a236c85aa364-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823673    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823709    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec043c-9364-4fa0-a313-a236c85aa364-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823747    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824015 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.823815    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824327 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.824007    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824508 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.824480    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90ec043c-9364-4fa0-a313-a236c85aa364-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824615 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.824580    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824615 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.824602    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.824817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.824781    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.826180 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.826154    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90ec043c-9364-4fa0-a313-a236c85aa364-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.827104 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.827076    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90ec043c-9364-4fa0-a313-a236c85aa364-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.876826 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.876787    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.876978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.876908    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthlb\" (UniqueName: \"kubernetes.io/projected/90ec043c-9364-4fa0-a313-a236c85aa364-kube-api-access-jthlb\") pod \"maas-default-gateway-openshift-default-7f5667f86c-lxmnf\" (UID: \"90ec043c-9364-4fa0-a313-a236c85aa364\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:10.981125 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:10.981045    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:11.438070 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.438041    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"]
Nov 03 12:55:11.439171 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:55:11.439144    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec043c_9364_4fa0_a313_a236c85aa364.slice/crio-aabdb666aa2c9d9967177cd1b2a0c7c19492f25b31e6f79ee3c114dfbf4fa760 WatchSource:0}: Error finding container aabdb666aa2c9d9967177cd1b2a0c7c19492f25b31e6f79ee3c114dfbf4fa760: Status 404 returned error can't find the container with id aabdb666aa2c9d9967177cd1b2a0c7c19492f25b31e6f79ee3c114dfbf4fa760
Nov 03 12:55:11.442322 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.442288    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:55:11.442440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.442354    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:55:11.442440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.442433    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 12:55:11.707899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.707808    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" event={"ID":"90ec043c-9364-4fa0-a313-a236c85aa364","Type":"ContainerStarted","Data":"b9cee4ee7c32cdcdb023f27ea437a35d2f027e95b72cdee846b3a562b81a2155"}
Nov 03 12:55:11.707899 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.707889    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" event={"ID":"90ec043c-9364-4fa0-a313-a236c85aa364","Type":"ContainerStarted","Data":"aabdb666aa2c9d9967177cd1b2a0c7c19492f25b31e6f79ee3c114dfbf4fa760"}
Nov 03 12:55:11.818250 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.818187    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podStartSLOduration=1.8181663019999998 podStartE2EDuration="1.818166302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:55:11.81514808 +0000 UTC m=+3818.257350385" watchObservedRunningTime="2026-11-03 12:55:11.818166302 +0000 UTC m=+3818.260368608"
Nov 03 12:55:11.981568 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:11.981491    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:12.399849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:12.399810    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:12.983127 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:12.983093    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:12.983563 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:12.983148    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:13.981566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:13.981522    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:13.981810 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:13.981583    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:14.983171 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:14.983128    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:14.983704 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:14.983191    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:15.981824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:15.981790    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:15.982012 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:15.981840    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:16.898655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:16.898623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:16.981356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:16.981321    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:16.981554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:16.981431    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:17.982010 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:17.981975    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:17.982475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:17.982044    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:18.981877 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:18.981837    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Nov 03 12:55:18.982092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:18.981903    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Nov 03 12:55:20.258766 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:20.258727    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:20.259335 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:20.259310    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:20.260400 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:20.260358    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf"
Nov 03 12:55:21.415422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:21.415387    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:25.504789 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.504746    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 12:55:25.508937 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.508913    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.597454 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.597417    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjrh\" (UniqueName: \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.597609 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.597536    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.597697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.597677    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.597749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.597734    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.604618 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.604595    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b737fe-serving-cert\""
Nov 03 12:55:25.607978 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.607957    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 12:55:25.700817 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.700779    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjrh\" (UniqueName: \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.701019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.700857    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.701019 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.700976    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.701128 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.701048    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.703000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.702966    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.703135 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:55:25.703097    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-serving-cert: secret "maas-api-e2e-extmod-a-b737fe-serving-cert" not found
Nov 03 12:55:25.703197 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:55:25.703173    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls podName:cdc93774-303d-4a1a-b3e0-4d9c9298fbce nodeName:}" failed. No retries permitted until 2026-11-03 12:55:26.203155022 +0000 UTC m=+3832.645357319 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls") pod "maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" (UID: "cdc93774-303d-4a1a-b3e0-4d9c9298fbce") : secret "maas-api-e2e-extmod-a-b737fe-serving-cert" not found
Nov 03 12:55:25.707444 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.707419    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.715063 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.715033    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:25.756913 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.756829    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjrh\" (UniqueName: \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:25.876836 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:25.876803    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:26.100783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.100677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:26.206643 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.206609    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:26.209098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.209071    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:26.428090 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.428054    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:26.478889 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:55:26.478848    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:55:26.848098 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.848072    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 12:55:26.849644 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:55:26.849619    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc93774_303d_4a1a_b3e0_4d9c9298fbce.slice/crio-2b2ed6196e77483612f1c7f31242994dab848b284bac791d98b31b0f810f0178 WatchSource:0}: Error finding container 2b2ed6196e77483612f1c7f31242994dab848b284bac791d98b31b0f810f0178: Status 404 returned error can't find the container with id 2b2ed6196e77483612f1c7f31242994dab848b284bac791d98b31b0f810f0178
Nov 03 12:55:26.850885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:26.850862    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 12:55:27.852204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:27.852170    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" event={"ID":"cdc93774-303d-4a1a-b3e0-4d9c9298fbce","Type":"ContainerStarted","Data":"49ea194d3d85f0246d2482c10698cb5d549a30dad8cac86ac3ccc1030da11cd3"}
Nov 03 12:55:27.852653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:27.852238    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" event={"ID":"cdc93774-303d-4a1a-b3e0-4d9c9298fbce","Type":"ContainerStarted","Data":"2b2ed6196e77483612f1c7f31242994dab848b284bac791d98b31b0f810f0178"}
Nov 03 12:55:27.852653 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:27.852397    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:27.988035 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:27.987981    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" podStartSLOduration=2.065696272 podStartE2EDuration="2.987968231s" totalImagesPullingTime="922.271959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:55:27.98444597 +0000 UTC m=+3834.426648294" watchObservedRunningTime="2026-11-03 12:55:27.987968231 +0000 UTC m=+3834.430170535"
Nov 03 12:55:30.047106 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:30.047073    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:30.278307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:30.278275    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:31.443479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:31.443441    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:33.868126 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:33.868097    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 12:55:34.721588 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:34.721549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:39.132226 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:39.132188    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:43.610716 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:43.610678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:47.978302 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:47.978267    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:50.545771 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:50.545740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:50.752968 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:50.752932    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:52.391870 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:52.391831    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:55:56.472509 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:55:56.472475    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:55:56.729161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:55:56.729075    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:01.217538 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:01.217500    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:01.670708 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:01.670669    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:05.297541 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.297484    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 12:56:05.303307 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.303287    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.399530 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.399490    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d126a6-serving-cert\""
Nov 03 12:56:05.402887 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.402864    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 12:56:05.478921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.478879    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.479137 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.479055    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrzf\" (UniqueName: \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.479204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.479148    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.479204 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.479182    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.495554 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.495524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:05.580508 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.580358    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrzf\" (UniqueName: \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.580508 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.580440    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.580508 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.580472    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.580828 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.580517    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.581435 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.581373    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.583083 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.583060    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.583214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.583149    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.633097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.633063    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrzf\" (UniqueName: \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\") pod \"maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:05.653780 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.653746    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:05.802885 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.802850    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:05.918024 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:05.917995    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:06.283777 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:06.283746    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 12:56:07.139911 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:07.139877    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" event={"ID":"8044ffc2-7a94-4cc9-a05e-68d3af009280","Type":"ContainerStarted","Data":"5da7f0f1cb8426978e9d3e029d7c27c5336e042850878994ea32a184e8e62b4d"}
Nov 03 12:56:08.151366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:08.151335    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" event={"ID":"8044ffc2-7a94-4cc9-a05e-68d3af009280","Type":"ContainerStarted","Data":"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"}
Nov 03 12:56:08.151786 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:08.151443    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:08.261072 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:08.261006    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" podStartSLOduration=2.36600837 podStartE2EDuration="3.260971832s" totalImagesPullingTime="894.963462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:56:08.257183681 +0000 UTC m=+3874.699386001" watchObservedRunningTime="2026-11-03 12:56:08.260971832 +0000 UTC m=+3874.703174138"
Nov 03 12:56:10.037844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:10.037799    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:14.163976 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:14.163896    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 12:56:14.524629 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:14.524532    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:18.998518 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:18.998480    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:23.600099 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:23.600062    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:26.472823 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:56:26.472775    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:56:28.105827 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:28.105778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:30.549351 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:30.549310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:30.706586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:30.706551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:32.555134 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:32.555096    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:34.389000 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:34.388966    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:56:34.410974 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:34.410950    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 12:56:37.004522 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:37.004475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:41.412052 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:41.412015    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:43.783175 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:43.783129    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:43.965058 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:43.965019    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:45.784639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:45.784596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:45.934236 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:45.934207    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:46.370440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:46.370402    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:46.816713 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:46.816672    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:50.492744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:50.492708    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:50.802622 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:50.802536    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:50.969445 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:50.969401    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:51.140118 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:51.140086    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:55.272592 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:55.272557    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:55.428895 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:55.428862    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:55.807412 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:55.807354    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:56.473659 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:56:56.473622    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:56:57.132731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:57.132676    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:57.943525 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:57.943351    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:58.109332 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:58.109273    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:56:59.654765 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:56:59.654732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:01.795439 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:01.795403    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:01.957731 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:01.957698    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:02.121571 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:02.121475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:04.161172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:04.161138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:04.339133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:04.339099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:04.817112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:04.817065    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:04.987783 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:04.987753    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:08.932356 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:08.932279    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:09.197460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:09.197273    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 12:57:09.197460 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:09.197421    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 12:57:13.330988 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:13.330957    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:17.720314 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:17.720268    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:22.110110 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:22.110074    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:26.473342 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:57:26.473293    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:57:26.579446 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:26.579406    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:31.012654 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:31.012617    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:35.716570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:35.716470    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:36.341871 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:36.341825    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:36.761591 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:36.761555    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:37.350680 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:37.350627    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:37.828366 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:37.828329    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:40.231368 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:40.231328    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:41.443178 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:41.443095    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:41.610824 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:41.610795    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:41.771535 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:41.771449    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:44.474896 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:44.474865    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:44.636131 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:44.636098    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:45.013202 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.013168    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 12:57:45.017586 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.017562    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.069633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.069579    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 12:57:45.070639 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.069594    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-aa9221\""
Nov 03 12:57:45.106411 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.106339    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:45.111595 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.111577    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-aa9221-dockercfg-x4gmf\""
Nov 03 12:57:45.181873 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.181842    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krk7d\" (UniqueName: \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.182090 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.181905    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.283941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.283852    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krk7d\" (UniqueName: \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.284112 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.284016    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.284776 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.284748    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.336453 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.336421    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krk7d\" (UniqueName: \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\") pod \"payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") " pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:45.632636 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:45.632603    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:46.001577 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:46.001547    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 12:57:46.002604 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:57:46.002578    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3066862a_c259_4bac_96a9_9da266214166.slice/crio-6c2ae279496283b50c85da6926741274ce6acba3f987280077d49de40bf34e3a WatchSource:0}: Error finding container 6c2ae279496283b50c85da6926741274ce6acba3f987280077d49de40bf34e3a: Status 404 returned error can't find the container with id 6c2ae279496283b50c85da6926741274ce6acba3f987280077d49de40bf34e3a
Nov 03 12:57:46.974990 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:46.974953    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" event={"ID":"3066862a-c259-4bac-96a9-9da266214166","Type":"ContainerStarted","Data":"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"}
Nov 03 12:57:46.975536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:46.975009    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" event={"ID":"3066862a-c259-4bac-96a9-9da266214166","Type":"ContainerStarted","Data":"6c2ae279496283b50c85da6926741274ce6acba3f987280077d49de40bf34e3a"}
Nov 03 12:57:46.975536 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:46.975170    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:47.083722 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:47.083653    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" podStartSLOduration=3.083633991 podStartE2EDuration="3.083633991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:57:47.081628809 +0000 UTC m=+3973.523831134" watchObservedRunningTime="2026-11-03 12:57:47.083633991 +0000 UTC m=+3973.525836298"
Nov 03 12:57:49.838888 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:49.838851    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:54.395287 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:54.395255    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:57:56.472846 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:57:56.472805    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:57:57.985354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:57.985323    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 12:57:58.811940 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:57:58.811899    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:02.350531 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:02.350486    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:03.229172 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:03.229076    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:07.609111 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:07.609075    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:11.670792 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.670755    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 12:58:11.675133 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.675110    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.770113 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.770076    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-532442-serving-cert\""
Nov 03 12:58:11.773572 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.773552    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 12:58:11.787675 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.787646    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.787816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.787725    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.787816 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.787779    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkrt\" (UniqueName: \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.787910 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.787892    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.889560 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.889528    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.889749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.889582    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.889749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.889642    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.889749 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.889686    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkrt\" (UniqueName: \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.889917 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:11.889683    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-serving-cert: secret "maas-api-e2e-inf-a-532442-serving-cert" not found
Nov 03 12:58:11.889917 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:11.889885    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls podName:ca4624e9-0305-4be3-85f9-860c6da98e7d nodeName:}" failed. No retries permitted until 2026-11-03 12:58:12.389860087 +0000 UTC m=+3998.832062385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls") pod "maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" (UID: "ca4624e9-0305-4be3-85f9-860c6da98e7d") : secret "maas-api-e2e-inf-a-532442-serving-cert" not found
Nov 03 12:58:11.890354 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.890334    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.892138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.892117    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:11.941495 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.941388    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:11.943237 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:11.943218    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkrt\" (UniqueName: \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:12.088693 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.088650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:12.239617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.239531    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:12.395633 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.395588    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:12.398023 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.397993    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") pod \"maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:12.590737 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.590645    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:12.962419 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:12.962372    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 12:58:12.963096 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:58:12.963072    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4624e9_0305_4be3_85f9_860c6da98e7d.slice/crio-76168e38910e35e0792f0d764444e06cadd8047157fb0dccc2bbdcf9844da354 WatchSource:0}: Error finding container 76168e38910e35e0792f0d764444e06cadd8047157fb0dccc2bbdcf9844da354: Status 404 returned error can't find the container with id 76168e38910e35e0792f0d764444e06cadd8047157fb0dccc2bbdcf9844da354
Nov 03 12:58:13.167815 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:13.167779    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" event={"ID":"ca4624e9-0305-4be3-85f9-860c6da98e7d","Type":"ContainerStarted","Data":"76168e38910e35e0792f0d764444e06cadd8047157fb0dccc2bbdcf9844da354"}
Nov 03 12:58:14.176440 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:14.176401    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" event={"ID":"ca4624e9-0305-4be3-85f9-860c6da98e7d","Type":"ContainerStarted","Data":"434e221d3ae8f3ba7d8af16fa9fb1b231e6fd44105bb2bbdc9eb05cab17f1363"}
Nov 03 12:58:14.176941 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:14.176475    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:14.285067 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:14.284963    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" podStartSLOduration=2.276915621 podStartE2EDuration="3.284945418s" totalImagesPullingTime="1.008029797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:58:14.281785082 +0000 UTC m=+4000.723987387" watchObservedRunningTime="2026-11-03 12:58:14.284945418 +0000 UTC m=+4000.727147722"
Nov 03 12:58:16.500518 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:16.500478    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:20.190365 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:20.190335    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 12:58:20.855679 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:20.855647    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:25.230123 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:25.230077    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:26.474314 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:26.474278    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:58:29.383570 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.383534    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 12:58:29.388994 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.388972    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.394030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.394005    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.394148 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.394044    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxrb2\" (UniqueName: \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.394214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.394130    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.394214 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.394199    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.441438 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.441408    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 12:58:29.482712 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.482678    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-fb74b5-serving-cert\""
Nov 03 12:58:29.495479 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.495454    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.495602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.495488    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxrb2\" (UniqueName: \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.495602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.495512    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.495602 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.495560    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.496066 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:29.495612    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-serving-cert: secret "maas-api-e2e-extmod-b-fb74b5-serving-cert" not found
Nov 03 12:58:29.496066 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:29.495689    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls podName:e0c22382-ce6b-43c7-af2c-55fa7c6ca00e nodeName:}" failed. No retries permitted until 2026-11-03 12:58:29.995670106 +0000 UTC m=+4016.437872394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls") pod "maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" (UID: "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e") : secret "maas-api-e2e-extmod-b-fb74b5-serving-cert" not found
Nov 03 12:58:29.496302 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.496283    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.498008 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.497988    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.556932 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.556897    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxrb2\" (UniqueName: \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:29.596860 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.596827    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:29.750928 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.750840    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:29.906115 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:29.906077    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:30.001446 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:30.001340    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:30.003843 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:30.003823    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") pod \"maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:30.303192 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:30.303100    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:30.679038 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:58:30.679007    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c22382_ce6b_43c7_af2c_55fa7c6ca00e.slice/crio-d8bcb258f5a414099ac0ffdbf31e4a5b29260cfaf22a243572eb82b4ed684931 WatchSource:0}: Error finding container d8bcb258f5a414099ac0ffdbf31e4a5b29260cfaf22a243572eb82b4ed684931: Status 404 returned error can't find the container with id d8bcb258f5a414099ac0ffdbf31e4a5b29260cfaf22a243572eb82b4ed684931
Nov 03 12:58:30.679740 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:30.679715    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 12:58:31.311587 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:31.311550    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" event={"ID":"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e","Type":"ContainerStarted","Data":"d8bcb258f5a414099ac0ffdbf31e4a5b29260cfaf22a243572eb82b4ed684931"}
Nov 03 12:58:32.321030 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:32.320993    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" event={"ID":"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e","Type":"ContainerStarted","Data":"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"}
Nov 03 12:58:32.321475 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:32.321212    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:32.430748 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:32.430684    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" podStartSLOduration=2.5449440819999998 podStartE2EDuration="3.430664798s" totalImagesPullingTime="885.720716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:58:32.426653654 +0000 UTC m=+4018.868855982" watchObservedRunningTime="2026-11-03 12:58:32.430664798 +0000 UTC m=+4018.872867135"
Nov 03 12:58:34.147559 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:34.147524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:38.335655 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:38.335622    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 12:58:38.534844 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:38.534795    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:42.924231 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:42.924201    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:47.318082 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:47.318046    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:51.730976 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:51.730940    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:53.124328 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:53.124296    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:58:53.124797 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:53.124512    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" containerID="cri-o://034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428" gracePeriod=30
Nov 03 12:58:55.302357 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:55.302315    5784 patch_prober.go:29] interesting pod/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 12:58:55.302774 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:55.302371    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 12:58:56.130132 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:56.130088    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:58:56.473443 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:56.473342    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:58:58.380562 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.380534    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:58:58.456427 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456311    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.456659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456441    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.456659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456513    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.456659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456541    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.456659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456588    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.456659 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456648    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.457073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456735    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.457073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456782    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.457073 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.456829    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\") pod \"f2c21463-38c8-4195-9211-d258590fe08f\" (UID: \"f2c21463-38c8-4195-9211-d258590fe08f\") "
Nov 03 12:58:58.457317 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457068    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:58:58.457317 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457183    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:58:58.457510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457432    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.457510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457454    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.457703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457526    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:58:58.457703 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.457587    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:58:58.459393 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.459214    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "kube-api-access-n8hnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:58:58.459510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.459428    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 12:58:58.459510 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.459480    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 12:58:58.459921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.459891    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 12:58:58.459921 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.459893    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy" pod "f2c21463-38c8-4195-9211-d258590fe08f" (UID: "f2c21463-38c8-4195-9211-d258590fe08f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 12:58:58.511388 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.511337    5784 generic.go:356] "Generic (PLEG): container finished" podID="f2c21463-38c8-4195-9211-d258590fe08f" containerID="034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428" exitCode=0
Nov 03 12:58:58.511566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.511417    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"
Nov 03 12:58:58.511566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.511423    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" event={"ID":"f2c21463-38c8-4195-9211-d258590fe08f","Type":"ContainerDied","Data":"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"}
Nov 03 12:58:58.511566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.511482    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r" event={"ID":"f2c21463-38c8-4195-9211-d258590fe08f","Type":"ContainerDied","Data":"c8bbcde0d2c06255a9477993c90a0f3cc94c41f0c4c8bfbbdb17b82b29779f1c"}
Nov 03 12:58:58.511566 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.511505    5784 scope.go:122] "RemoveContainer" containerID="034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"
Nov 03 12:58:58.531097 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.531069    5784 scope.go:122] "RemoveContainer" containerID="034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"
Nov 03 12:58:58.533791 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:58:58.533763    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428\": container with ID starting with 034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428 not found: ID does not exist" containerID="034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"
Nov 03 12:58:58.533919 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.533801    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428"} err="failed to get container status \"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428\": rpc error: code = NotFound desc = could not find container \"034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428\": container with ID starting with 034e6958825c83c89cf9e710dd220a9c931313ea9a98a28a6b523ec8e2a82428 not found: ID does not exist"
Nov 03 12:58:58.559020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.558981    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559020 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559015    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8hnm\" (UniqueName: \"kubernetes.io/projected/f2c21463-38c8-4195-9211-d258590fe08f-kube-api-access-n8hnm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559032    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c21463-38c8-4195-9211-d258590fe08f-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559047    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c21463-38c8-4195-9211-d258590fe08f-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559061    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559075    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.559232 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.559089    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c21463-38c8-4195-9211-d258590fe08f-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 12:58:58.669288 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.669228    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:58:58.672879 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:58.672851    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-6fb3a8-openshift-default-6bcc9c6955-gjt2r"]
Nov 03 12:58:59.682849 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:58:59.682816    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c21463-38c8-4195-9211-d258590fe08f" path="/var/lib/kubelet/pods/f2c21463-38c8-4195-9211-d258590fe08f/volumes"
Nov 03 12:59:00.500733 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:00.500693    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:02.655581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:02.655537    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:04.882138 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:04.882101    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:08.972966 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:08.972931    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 12:59:08.973685 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:08.973666    5784 cpu_manager.go:414] "containerMap: removing container" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy"
Nov 03 12:59:08.973744 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:08.973686    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy"
Nov 03 12:59:08.973806 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:08.973796    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c21463-38c8-4195-9211-d258590fe08f" containerName="istio-proxy"
Nov 03 12:59:08.977034 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:08.977015    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.026323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.026270    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aa9221-serving-cert\""
Nov 03 12:59:09.075723 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.075680    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 12:59:09.092161 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.092127    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.092323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.092188    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.092323 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.092275    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jwb\" (UniqueName: \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.092443 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.092329    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.180961 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.180927    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:09.193295 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.193267    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.193464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.193322    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.193464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.193359    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jwb\" (UniqueName: \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.193464 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.193424    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.193629 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:59:09.193541    5784 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-serving-cert: secret "maas-api-e2e-inf-b-aa9221-serving-cert" not found
Nov 03 12:59:09.193629 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:59:09.193615    5784 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls podName:bcf93bee-986a-41ed-82a9-8230a01d3ef9 nodeName:}" failed. No retries permitted until 2026-11-03 12:59:09.69359127 +0000 UTC m=+4056.135793552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls") pod "maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" (UID: "bcf93bee-986a-41ed-82a9-8230a01d3ef9") : secret "maas-api-e2e-inf-b-aa9221-serving-cert" not found
Nov 03 12:59:09.194037 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.194015    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.195697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.195673    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.247638 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.247556    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jwb\" (UniqueName: \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.330388 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.330342    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:09.477803 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.477766    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:09.699092 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.699058    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.701697 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.701665    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") pod \"maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:09.890545 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:09.890502    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:10.259313 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:10.259277    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 12:59:10.259718 ip-10-0-139-140 kubenswrapper[5784]: W1103 12:59:10.259556    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf93bee_986a_41ed_82a9_8230a01d3ef9.slice/crio-0cfe22e46c10b8ea2aaa6a4181d81c64a78b34ec83149e644bffb6606e5e57ac WatchSource:0}: Error finding container 0cfe22e46c10b8ea2aaa6a4181d81c64a78b34ec83149e644bffb6606e5e57ac: Status 404 returned error can't find the container with id 0cfe22e46c10b8ea2aaa6a4181d81c64a78b34ec83149e644bffb6606e5e57ac
Nov 03 12:59:10.601882 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:10.601758    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" event={"ID":"bcf93bee-986a-41ed-82a9-8230a01d3ef9","Type":"ContainerStarted","Data":"0cfe22e46c10b8ea2aaa6a4181d81c64a78b34ec83149e644bffb6606e5e57ac"}
Nov 03 12:59:11.610581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:11.610481    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" event={"ID":"bcf93bee-986a-41ed-82a9-8230a01d3ef9","Type":"ContainerStarted","Data":"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"}
Nov 03 12:59:11.610581 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:11.610559    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:11.719858 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:11.719790    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" podStartSLOduration=2.83808877 podStartE2EDuration="3.719772073s" totalImagesPullingTime="881.683303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 12:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 12:59:11.716491383 +0000 UTC m=+4058.158693690" watchObservedRunningTime="2026-11-03 12:59:11.719772073 +0000 UTC m=+4058.161974382"
Nov 03 12:59:13.863071 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:13.863035    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:17.624272 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:17.624242    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 12:59:18.236258 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:18.236219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:22.625993 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:22.625963    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:26.473956 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:59:26.473918    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:59:27.120833 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:27.120792    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:31.530360 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:31.530326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:35.948737 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:35.948694    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:40.329422 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:40.329370    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:44.733808 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:44.733777    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:49.227310 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:49.227274    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:53.950617 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:53.950586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:54.361001 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:54.360913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 12:59:56.473724 ip-10-0-139-140 kubenswrapper[5784]: E1103 12:59:56.473684    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 12:59:58.327413 ip-10-0-139-140 kubenswrapper[5784]: I1103 12:59:58.327353    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:00.043554 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:00.043516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:00.891167 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:00.891135    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:01.136367 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:01.136331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:01.296583 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:01.296500    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:02.859678 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:02.859641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:03.005775 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:03.005740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:06.567146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:06.567108    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:07.533281 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:07.533246    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:07.708036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:07.708003    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:11.312317 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:11.312271    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:11.468620 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:11.468579    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:12.377553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:12.377517    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:14.030923 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:14.030888    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:16.812179 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:16.812147    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:19.528392 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:19.528345    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:19.691659 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:19.691630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:21.324363 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:21.324314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:21.617428 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:21.617320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:23.226323 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:23.226285    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:25.714500 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:25.714470    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:26.473202 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:00:26.473163    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:00:27.183584 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:27.183549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:27.369144 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:27.369112    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:30.178330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:30.178288    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:32.407658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:32.407621    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:32.589394 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:32.589320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:33.471312 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:33.471273    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:34.668602 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:34.668572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:38.998765 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:38.998728    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:39.197687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:39.197653    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:43.441596 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:43.441498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:45.274645 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:45.274590    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:45.438193 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:45.438160    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:47.914620 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:47.914589    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:51.193797 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:51.193764    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:51.960352 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:51.960314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:52.116984 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:52.116949    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:52.550985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:52.550952    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:56.473118 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:00:56.473083    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:00:56.581692 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:56.581661    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:00:57.365258 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:00:57.365227    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:01.756765 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:01.756733    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:03.286938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:03.286899    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:03.819722 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:03.819684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:03.975282 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:03.975235    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:06.124359 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:06.124322    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:10.645593 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:10.645552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:14.331968 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.331925    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 13:01:14.332469 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.332198    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" podUID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" containerName="maas-api" containerID="cri-o://49ea194d3d85f0246d2482c10698cb5d549a30dad8cac86ac3ccc1030da11cd3" gracePeriod=30
Nov 03 13:01:14.455952 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.455918    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:14.540515 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.521565    5784 generic.go:356] "Generic (PLEG): container finished" podID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" containerID="49ea194d3d85f0246d2482c10698cb5d549a30dad8cac86ac3ccc1030da11cd3" exitCode=0
Nov 03 13:01:14.540515 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.521659    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" event={"ID":"cdc93774-303d-4a1a-b3e0-4d9c9298fbce","Type":"ContainerDied","Data":"49ea194d3d85f0246d2482c10698cb5d549a30dad8cac86ac3ccc1030da11cd3"}
Nov 03 13:01:14.632274 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.631957    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 13:01:14.656643 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.656602    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") pod \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") "
Nov 03 13:01:14.656849 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.656699    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\" (UniqueName: \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\") pod \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") "
Nov 03 13:01:14.656849 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.656799    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\") pod \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") "
Nov 03 13:01:14.656975 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.656877    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\") pod \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\" (UID: \"cdc93774-303d-4a1a-b3e0-4d9c9298fbce\") "
Nov 03 13:01:14.657498 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.657446    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca" pod "cdc93774-303d-4a1a-b3e0-4d9c9298fbce" (UID: "cdc93774-303d-4a1a-b3e0-4d9c9298fbce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:14.660356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.659748    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh" pod "cdc93774-303d-4a1a-b3e0-4d9c9298fbce" (UID: "cdc93774-303d-4a1a-b3e0-4d9c9298fbce"). InnerVolumeSpecName "kube-api-access-4tjrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:14.660356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.660124    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls" pod "cdc93774-303d-4a1a-b3e0-4d9c9298fbce" (UID: "cdc93774-303d-4a1a-b3e0-4d9c9298fbce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:14.661154 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.661129    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs" pod "cdc93774-303d-4a1a-b3e0-4d9c9298fbce" (UID: "cdc93774-303d-4a1a-b3e0-4d9c9298fbce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:14.758568 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.758532    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:14.758799 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.758781    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tjrh\" (UniqueName: \"kubernetes.io/projected/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-kube-api-access-4tjrh\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:14.758919 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.758905    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:14.759014 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:14.759003    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdc93774-303d-4a1a-b3e0-4d9c9298fbce-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:15.394662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.394624    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 13:01:15.395134 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.394872    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" podUID="ca4624e9-0305-4be3-85f9-860c6da98e7d" containerName="maas-api" containerID="cri-o://434e221d3ae8f3ba7d8af16fa9fb1b231e6fd44105bb2bbdc9eb05cab17f1363" gracePeriod=30
Nov 03 13:01:15.517333 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.517301    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:15.533662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.533622    5784 generic.go:356] "Generic (PLEG): container finished" podID="ca4624e9-0305-4be3-85f9-860c6da98e7d" containerID="434e221d3ae8f3ba7d8af16fa9fb1b231e6fd44105bb2bbdc9eb05cab17f1363" exitCode=0
Nov 03 13:01:15.533821 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.533802    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" event={"ID":"ca4624e9-0305-4be3-85f9-860c6da98e7d","Type":"ContainerDied","Data":"434e221d3ae8f3ba7d8af16fa9fb1b231e6fd44105bb2bbdc9eb05cab17f1363"}
Nov 03 13:01:15.535629 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.535607    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"
Nov 03 13:01:15.536553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.535603    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd" event={"ID":"cdc93774-303d-4a1a-b3e0-4d9c9298fbce","Type":"ContainerDied","Data":"2b2ed6196e77483612f1c7f31242994dab848b284bac791d98b31b0f810f0178"}
Nov 03 13:01:15.536663 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.536594    5784 scope.go:122] "RemoveContainer" containerID="49ea194d3d85f0246d2482c10698cb5d549a30dad8cac86ac3ccc1030da11cd3"
Nov 03 13:01:15.707794 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.707767    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 13:01:15.715859 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.715833    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 13:01:15.721456 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.721429    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b737fe-5566bc7cfd-vcdpd"]
Nov 03 13:01:15.769800 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.769764    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\") pod \"ca4624e9-0305-4be3-85f9-860c6da98e7d\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") "
Nov 03 13:01:15.770156 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.770128    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca" pod "ca4624e9-0305-4be3-85f9-860c6da98e7d" (UID: "ca4624e9-0305-4be3-85f9-860c6da98e7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:15.770156 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.770146    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\" (UniqueName: \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\") pod \"ca4624e9-0305-4be3-85f9-860c6da98e7d\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") "
Nov 03 13:01:15.770330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.770192    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") pod \"ca4624e9-0305-4be3-85f9-860c6da98e7d\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") "
Nov 03 13:01:15.770330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.770313    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\") pod \"ca4624e9-0305-4be3-85f9-860c6da98e7d\" (UID: \"ca4624e9-0305-4be3-85f9-860c6da98e7d\") "
Nov 03 13:01:15.781166 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.771238    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca4624e9-0305-4be3-85f9-860c6da98e7d-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:15.781166 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.772480    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt" pod "ca4624e9-0305-4be3-85f9-860c6da98e7d" (UID: "ca4624e9-0305-4be3-85f9-860c6da98e7d"). InnerVolumeSpecName "kube-api-access-kzkrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:15.781166 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.773099    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls" pod "ca4624e9-0305-4be3-85f9-860c6da98e7d" (UID: "ca4624e9-0305-4be3-85f9-860c6da98e7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:15.781807 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.781561    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs" pod "ca4624e9-0305-4be3-85f9-860c6da98e7d" (UID: "ca4624e9-0305-4be3-85f9-860c6da98e7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:15.875962 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.875925    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:15.875962 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.875950    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca4624e9-0305-4be3-85f9-860c6da98e7d-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:15.875962 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:15.875961    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkrt\" (UniqueName: \"kubernetes.io/projected/ca4624e9-0305-4be3-85f9-860c6da98e7d-kube-api-access-kzkrt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:16.553512 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.549253    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"
Nov 03 13:01:16.553512 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.549280    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm" event={"ID":"ca4624e9-0305-4be3-85f9-860c6da98e7d","Type":"ContainerDied","Data":"76168e38910e35e0792f0d764444e06cadd8047157fb0dccc2bbdcf9844da354"}
Nov 03 13:01:16.553512 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.549351    5784 scope.go:122] "RemoveContainer" containerID="434e221d3ae8f3ba7d8af16fa9fb1b231e6fd44105bb2bbdc9eb05cab17f1363"
Nov 03 13:01:16.702144 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.702092    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 13:01:16.705227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.705182    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-532442-5cc88f58c8-kc9wm"]
Nov 03 13:01:16.795648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.795619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:16.947041 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:16.947003    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:17.172611 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.172579    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:17.339195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.339106    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:17.504927 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.504885    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:17.682531 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.682499    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4624e9-0305-4be3-85f9-860c6da98e7d" path="/var/lib/kubelet/pods/ca4624e9-0305-4be3-85f9-860c6da98e7d/volumes"
Nov 03 13:01:17.682967 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.682954    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" path="/var/lib/kubelet/pods/cdc93774-303d-4a1a-b3e0-4d9c9298fbce/volumes"
Nov 03 13:01:17.944239 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:17.944160    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:18.104198 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:18.104167    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:19.332024 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:19.331988    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:20.038001 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:20.037968    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:20.762370 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:20.762338    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:21.817166 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:21.817135    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:23.181225 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:23.181187    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:23.344096 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:23.344064    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:23.684646 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:23.684612    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:23.854291 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:23.854259    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:24.616259 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:24.616217    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:25.090915 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:25.090884    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:26.473547 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:01:26.473507    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:01:26.667266 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:26.667222    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:27.062513 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:27.062466    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:27.583553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:27.583502    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:27.750636 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:27.750603    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:29.697121 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:29.697070    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:29.882572 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:29.882538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:30.126620 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:30.126586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:30.660425 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:30.660358    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:30.720665 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:30.720629    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 13:01:30.721094 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:30.720932    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" podUID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" containerName="maas-api" containerID="cri-o://c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d" gracePeriod=30
Nov 03 13:01:30.845154 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:30.845115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:31.011389 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.011350    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 13:01:31.065213 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.065182    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\") pod \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") "
Nov 03 13:01:31.065417 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.065320    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") pod \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") "
Nov 03 13:01:31.065480 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.065424    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\" (UniqueName: \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\") pod \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") "
Nov 03 13:01:31.065480 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.065452    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\") pod \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\" (UID: \"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e\") "
Nov 03 13:01:31.066028 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.066003    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca" pod "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" (UID: "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:31.067435 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.067407    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs" pod "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" (UID: "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:31.067847 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.067822    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls" pod "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" (UID: "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:31.068053 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.068037    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2" pod "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" (UID: "e0c22382-ce6b-43c7-af2c-55fa7c6ca00e"). InnerVolumeSpecName "kube-api-access-sxrb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:31.167464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.167422    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:31.167464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.167461    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxrb2\" (UniqueName: \"kubernetes.io/projected/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-kube-api-access-sxrb2\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:31.167710 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.167478    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:31.167710 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.167494    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:31.177762 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.177726    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:31.663764 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.663728    5784 generic.go:356] "Generic (PLEG): container finished" podID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" containerID="c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d" exitCode=0
Nov 03 13:01:31.663985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.663780    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"
Nov 03 13:01:31.663985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.663812    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" event={"ID":"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e","Type":"ContainerDied","Data":"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"}
Nov 03 13:01:31.663985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.663871    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf" event={"ID":"e0c22382-ce6b-43c7-af2c-55fa7c6ca00e","Type":"ContainerDied","Data":"d8bcb258f5a414099ac0ffdbf31e4a5b29260cfaf22a243572eb82b4ed684931"}
Nov 03 13:01:31.663985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.663895    5784 scope.go:122] "RemoveContainer" containerID="c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"
Nov 03 13:01:31.677743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.677696    5784 scope.go:122] "RemoveContainer" containerID="c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"
Nov 03 13:01:31.678058 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:01:31.678032    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d\": container with ID starting with c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d not found: ID does not exist" containerID="c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"
Nov 03 13:01:31.678138 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.678072    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d"} err="failed to get container status \"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d\": rpc error: code = NotFound desc = could not find container \"c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d\": container with ID starting with c27cdce0f612fb79c7b03dba2aabca66f7043baf467d8d8092366e9cf571a19d not found: ID does not exist"
Nov 03 13:01:31.819169 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.819125    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 13:01:31.865189 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:31.865155    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-fb74b5-b97957479-8lnwf"]
Nov 03 13:01:32.517016 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:32.516981    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:32.802541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:32.802457    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:32.954872 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:32.954833    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:33.683035 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:33.683001    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" path="/var/lib/kubelet/pods/e0c22382-ce6b-43c7-af2c-55fa7c6ca00e/volumes"
Nov 03 13:01:34.200235 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:34.200206    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:34.442049 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:34.442023    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:01:34.474801 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:34.474725    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:01:34.709202 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:34.709174    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:35.707744 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:35.707716    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:35.871447 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:35.871405    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:36.041901 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:36.041814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:36.219712 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:36.219676    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:37.443424 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:37.443360    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:38.935049 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:38.935019    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:39.096853 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:39.096814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:42.019069 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:42.019036    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:45.645954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:45.645869    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:46.585044 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:46.585010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:46.836456 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:46.832090    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:51.223920 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:51.223878    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:54.966921 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:54.966885    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 13:01:54.967421 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:54.967165    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" podUID="8044ffc2-7a94-4cc9-a05e-68d3af009280" containerName="maas-api" containerID="cri-o://f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc" gracePeriod=30
Nov 03 13:01:55.090007 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.089975    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:55.252664 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.252635    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 13:01:55.302340 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.302309    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\") pod \"8044ffc2-7a94-4cc9-a05e-68d3af009280\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") "
Nov 03 13:01:55.302340 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.302344    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\") pod \"8044ffc2-7a94-4cc9-a05e-68d3af009280\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") "
Nov 03 13:01:55.302590 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.302366    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\") pod \"8044ffc2-7a94-4cc9-a05e-68d3af009280\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") "
Nov 03 13:01:55.302590 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.302452    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\" (UniqueName: \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\") pod \"8044ffc2-7a94-4cc9-a05e-68d3af009280\" (UID: \"8044ffc2-7a94-4cc9-a05e-68d3af009280\") "
Nov 03 13:01:55.302806 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.302778    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca" pod "8044ffc2-7a94-4cc9-a05e-68d3af009280" (UID: "8044ffc2-7a94-4cc9-a05e-68d3af009280"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:01:55.303220 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.303196    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8044ffc2-7a94-4cc9-a05e-68d3af009280-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:55.304877 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.304835    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls" pod "8044ffc2-7a94-4cc9-a05e-68d3af009280" (UID: "8044ffc2-7a94-4cc9-a05e-68d3af009280"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:55.304997 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.304893    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf" pod "8044ffc2-7a94-4cc9-a05e-68d3af009280" (UID: "8044ffc2-7a94-4cc9-a05e-68d3af009280"). InnerVolumeSpecName "kube-api-access-gjrzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:01:55.305182 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.305003    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs" pod "8044ffc2-7a94-4cc9-a05e-68d3af009280" (UID: "8044ffc2-7a94-4cc9-a05e-68d3af009280"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:01:55.405245 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.405203    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:55.405245 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.405245    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8044ffc2-7a94-4cc9-a05e-68d3af009280-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:55.405480 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.405261    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrzf\" (UniqueName: \"kubernetes.io/projected/8044ffc2-7a94-4cc9-a05e-68d3af009280-kube-api-access-gjrzf\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:01:55.895222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.895177    5784 generic.go:356] "Generic (PLEG): container finished" podID="8044ffc2-7a94-4cc9-a05e-68d3af009280" containerID="f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc" exitCode=0
Nov 03 13:01:55.895458 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.895242    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"
Nov 03 13:01:55.895458 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.895281    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" event={"ID":"8044ffc2-7a94-4cc9-a05e-68d3af009280","Type":"ContainerDied","Data":"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"}
Nov 03 13:01:55.895458 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.895352    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx" event={"ID":"8044ffc2-7a94-4cc9-a05e-68d3af009280","Type":"ContainerDied","Data":"5da7f0f1cb8426978e9d3e029d7c27c5336e042850878994ea32a184e8e62b4d"}
Nov 03 13:01:55.895458 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.895416    5784 scope.go:122] "RemoveContainer" containerID="f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"
Nov 03 13:01:55.906064 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.906035    5784 scope.go:122] "RemoveContainer" containerID="f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"
Nov 03 13:01:55.907460 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:01:55.907428    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc\": container with ID starting with f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc not found: ID does not exist" containerID="f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"
Nov 03 13:01:55.907603 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:55.907469    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc"} err="failed to get container status \"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc\": rpc error: code = NotFound desc = could not find container \"f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc\": container with ID starting with f8d65b1feb93287f0efc748fb3ddc31e0d2902d30431e3886b70b841ade383cc not found: ID does not exist"
Nov 03 13:01:56.051794 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:56.051704    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 13:01:56.054706 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:56.054669    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d126a6-6b6b467c98-f2stx"]
Nov 03 13:01:56.325411 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:56.325289    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:56.476765 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:01:56.476724    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:01:56.583460 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:56.583360    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:56.824913 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:56.824880    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:57.682165 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:57.682131    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8044ffc2-7a94-4cc9-a05e-68d3af009280" path="/var/lib/kubelet/pods/8044ffc2-7a94-4cc9-a05e-68d3af009280/volumes"
Nov 03 13:01:58.634178 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:58.634142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:01:58.881504 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:01:58.881463    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:01.363492 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.363438    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:01.681261 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.681229    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:01.840347 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.840314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:01.986724 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.986634    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:02:01.987204 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987190    5784 cpu_manager.go:414] "containerMap: removing container" podUID="8044ffc2-7a94-4cc9-a05e-68d3af009280" containerName="maas-api"
Nov 03 13:02:01.987204 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987203    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8044ffc2-7a94-4cc9-a05e-68d3af009280" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987244    5784 cpu_manager.go:414] "containerMap: removing container" podUID="ca4624e9-0305-4be3-85f9-860c6da98e7d" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987249    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4624e9-0305-4be3-85f9-860c6da98e7d" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987270    5784 cpu_manager.go:414] "containerMap: removing container" podUID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987275    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987292    5784 cpu_manager.go:414] "containerMap: removing container" podUID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" containerName="maas-api"
Nov 03 13:02:01.987356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987297    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" containerName="maas-api"
Nov 03 13:02:01.987727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987397    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc93774-303d-4a1a-b3e0-4d9c9298fbce" containerName="maas-api"
Nov 03 13:02:01.987727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987406    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="8044ffc2-7a94-4cc9-a05e-68d3af009280" containerName="maas-api"
Nov 03 13:02:01.987727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987423    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4624e9-0305-4be3-85f9-860c6da98e7d" containerName="maas-api"
Nov 03 13:02:01.987727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.987433    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0c22382-ce6b-43c7-af2c-55fa7c6ca00e" containerName="maas-api"
Nov 03 13:02:01.992670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:01.992647    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.039031 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.039000    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:02:02.045991 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.045691    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 13:02:02.045991 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.045757    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 13:02:02.045991 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.045782    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 13:02:02.046368 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.046334    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-rhbbs\""
Nov 03 13:02:02.048009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.047984    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:02.090978 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.090943    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091103    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091135    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091237 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091194    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091342 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091320    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8kf\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091421 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091360    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.091506 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.091493    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192489 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192452    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192489 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192493    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192740 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192620    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192740 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192666    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8kf\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192740 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192690    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192924 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192762    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.192924 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.192802    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.193364 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.193338    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.195144 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.195095    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.195267 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.195244    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.195491 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.195474    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.195609 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.195588    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.245133 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.245056    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.245429 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.245373    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8kf\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-c5qn2\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.311209 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.311174    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:02.734279 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:02:02.734250    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb373aa_4f44_4513_895c_1bbd96f1ccb4.slice/crio-5e2fdfb522f850714f2f386b43a1c77d62eea77efb5c13c31fcce18a772197a6 WatchSource:0}: Error finding container 5e2fdfb522f850714f2f386b43a1c77d62eea77efb5c13c31fcce18a772197a6: Status 404 returned error can't find the container with id 5e2fdfb522f850714f2f386b43a1c77d62eea77efb5c13c31fcce18a772197a6
Nov 03 13:02:02.735255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.735223    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:02:02.737172 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.737137    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:02:02.737288 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.737198    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:02:02.960658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.960616    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" event={"ID":"3eb373aa-4f44-4513-895c-1bbd96f1ccb4","Type":"ContainerStarted","Data":"220da0bc9f0543b318489ddd97c9d963165282d8cb8ff768230696b3385a688c"}
Nov 03 13:02:02.960984 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.960962    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" event={"ID":"3eb373aa-4f44-4513-895c-1bbd96f1ccb4","Type":"ContainerStarted","Data":"5e2fdfb522f850714f2f386b43a1c77d62eea77efb5c13c31fcce18a772197a6"}
Nov 03 13:02:02.962077 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:02.962053    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:03.073135 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.073015    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" podStartSLOduration=2.0730008030000002 podStartE2EDuration="2.073000803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:02:03.072442868 +0000 UTC m=+4229.514645173" watchObservedRunningTime="2026-11-03 13:02:03.073000803 +0000 UTC m=+4229.515203107"
Nov 03 13:02:03.139534 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.139501    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:03.617067 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.617029    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:03.782138 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.782103    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:03.971696 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.971664    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c5qn2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:02:03.971866 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:03.971707    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:02:04.354465 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:04.353362    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:04.807442 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:04.806946    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:04.963474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:04.963433    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:04.980686 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:04.980639    5784 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-c5qn2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 13:02:04.980892 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:04.980724    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 13:02:05.799456 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:05.799423    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:06.196358 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:06.196326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:07.980709 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:07.980678    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:02:09.984904 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:09.984858    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:10.235155 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:10.235054    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 13:02:10.578788 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:10.578681    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:10.732820 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:10.732786    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:11.021599 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:11.021553    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:11.177455 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:11.177333    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:13.665137 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:13.665102    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:15.235754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.235709    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" podUID="3066862a-c259-4bac-96a9-9da266214166" containerName="payload-processing" containerID="cri-o://7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0" gracePeriod=25
Nov 03 13:02:15.336493 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.336460    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:15.499887 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.499825    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 13:02:15.661444 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.661411    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\" (UniqueName: \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\") pod \"3066862a-c259-4bac-96a9-9da266214166\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") "
Nov 03 13:02:15.661620 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.661474    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\") pod \"3066862a-c259-4bac-96a9-9da266214166\" (UID: \"3066862a-c259-4bac-96a9-9da266214166\") "
Nov 03 13:02:15.661867 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.661845    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume" pod "3066862a-c259-4bac-96a9-9da266214166" (UID: "3066862a-c259-4bac-96a9-9da266214166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:02:15.663590 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.663567    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d" pod "3066862a-c259-4bac-96a9-9da266214166" (UID: "3066862a-c259-4bac-96a9-9da266214166"). InnerVolumeSpecName "kube-api-access-krk7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:02:15.762698 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.762613    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krk7d\" (UniqueName: \"kubernetes.io/projected/3066862a-c259-4bac-96a9-9da266214166-kube-api-access-krk7d\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:15.762698 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.762648    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3066862a-c259-4bac-96a9-9da266214166-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:02:15.771144 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:15.771107    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:16.066531 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.066434    5784 generic.go:356] "Generic (PLEG): container finished" podID="3066862a-c259-4bac-96a9-9da266214166" containerID="7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0" exitCode=0
Nov 03 13:02:16.066531 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.066485    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" event={"ID":"3066862a-c259-4bac-96a9-9da266214166","Type":"ContainerDied","Data":"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"}
Nov 03 13:02:16.066777 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.066567    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl" event={"ID":"3066862a-c259-4bac-96a9-9da266214166","Type":"ContainerDied","Data":"6c2ae279496283b50c85da6926741274ce6acba3f987280077d49de40bf34e3a"}
Nov 03 13:02:16.066777 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.066598    5784 scope.go:122] "RemoveContainer" containerID="7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"
Nov 03 13:02:16.066777 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.066765    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"
Nov 03 13:02:16.080106 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.080079    5784 scope.go:122] "RemoveContainer" containerID="7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"
Nov 03 13:02:16.081201 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:02:16.081169    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0\": container with ID starting with 7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0 not found: ID does not exist" containerID="7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"
Nov 03 13:02:16.081302 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.081213    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0"} err="failed to get container status \"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0\": rpc error: code = NotFound desc = could not find container \"7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0\": container with ID starting with 7654baa6efb4e5ef5cb2bff9c0f108d6ccb9b7b4b1d1a2f313fd82f78fef3fd0 not found: ID does not exist"
Nov 03 13:02:16.220601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.220566    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 13:02:16.265829 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:16.265796    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-aa9221-684b455b79-hw4jl"]
Nov 03 13:02:17.493457 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:17.493391    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:17.644612 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:17.644548    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:17.682733 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:17.682696    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3066862a-c259-4bac-96a9-9da266214166" path="/var/lib/kubelet/pods/3066862a-c259-4bac-96a9-9da266214166/volumes"
Nov 03 13:02:17.807657 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:17.807572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:19.446609 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:19.446559    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:20.005910 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:20.005870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:20.233328 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:20.233291    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:20.393877 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:20.393844    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:24.239153 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:24.239111    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:24.415153 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:24.415122    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:24.650687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:24.650645    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:24.840729 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:24.840694    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:26.473360 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:02:26.473323    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:02:27.398594 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:27.398558    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:27.558252 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:27.558214    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:29.905036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:29.904980    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:34.386347 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:34.386306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:39.002514 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:39.002479    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:43.534119 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:43.534076    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:48.033270 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:48.033233    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:52.596496 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:52.596460    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:56.478320 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:02:56.478277    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:02:57.031448 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:57.031405    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:02:57.210060 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:02:57.210002    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:01.637039 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:01.637010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:04.011572 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:04.011528    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:06.051917 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:06.051886    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:10.571685 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:10.571647    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:14.995442 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:14.995405    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:19.477886 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:19.477853    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:24.050176 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:24.050128    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:26.472152 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:03:26.472110    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:03:27.864190 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:27.864149    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 13:03:27.864693 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:27.864420    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" podUID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" containerName="maas-api" containerID="cri-o://81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267" gracePeriod=30
Nov 03 13:03:27.996637 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:27.996599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:28.160206 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.160179    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 13:03:28.254586 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.254554    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") pod \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") "
Nov 03 13:03:28.254789 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.254618    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\") pod \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") "
Nov 03 13:03:28.254789 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.254779    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\") pod \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") "
Nov 03 13:03:28.254970 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.254883    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\" (UniqueName: \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\") pod \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\" (UID: \"bcf93bee-986a-41ed-82a9-8230a01d3ef9\") "
Nov 03 13:03:28.257080 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.257046    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca" pod "bcf93bee-986a-41ed-82a9-8230a01d3ef9" (UID: "bcf93bee-986a-41ed-82a9-8230a01d3ef9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:03:28.258158 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.258133    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb" pod "bcf93bee-986a-41ed-82a9-8230a01d3ef9" (UID: "bcf93bee-986a-41ed-82a9-8230a01d3ef9"). InnerVolumeSpecName "kube-api-access-s5jwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:03:28.261521 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.261498    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls" pod "bcf93bee-986a-41ed-82a9-8230a01d3ef9" (UID: "bcf93bee-986a-41ed-82a9-8230a01d3ef9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:03:28.263404 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.263359    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs" pod "bcf93bee-986a-41ed-82a9-8230a01d3ef9" (UID: "bcf93bee-986a-41ed-82a9-8230a01d3ef9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:03:28.356951 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.356913    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:28.356951 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.356952    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf93bee-986a-41ed-82a9-8230a01d3ef9-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:28.357213 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.356967    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcf93bee-986a-41ed-82a9-8230a01d3ef9-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:28.357213 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.356983    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jwb\" (UniqueName: \"kubernetes.io/projected/bcf93bee-986a-41ed-82a9-8230a01d3ef9-kube-api-access-s5jwb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:03:28.649068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.649029    5784 generic.go:356] "Generic (PLEG): container finished" podID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" containerID="81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267" exitCode=0
Nov 03 13:03:28.649299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.649086    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"
Nov 03 13:03:28.649299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.649089    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" event={"ID":"bcf93bee-986a-41ed-82a9-8230a01d3ef9","Type":"ContainerDied","Data":"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"}
Nov 03 13:03:28.649299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.649221    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj" event={"ID":"bcf93bee-986a-41ed-82a9-8230a01d3ef9","Type":"ContainerDied","Data":"0cfe22e46c10b8ea2aaa6a4181d81c64a78b34ec83149e644bffb6606e5e57ac"}
Nov 03 13:03:28.649299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.649252    5784 scope.go:122] "RemoveContainer" containerID="81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"
Nov 03 13:03:28.665620 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.665599    5784 scope.go:122] "RemoveContainer" containerID="81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"
Nov 03 13:03:28.665905 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:03:28.665872    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267\": container with ID starting with 81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267 not found: ID does not exist" containerID="81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"
Nov 03 13:03:28.666048 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.665915    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267"} err="failed to get container status \"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267\": rpc error: code = NotFound desc = could not find container \"81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267\": container with ID starting with 81c65c2152720932f47317292a06cb9b1b111146ab1a688838bef2b2f6f1c267 not found: ID does not exist"
Nov 03 13:03:28.805428 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.805370    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 13:03:28.849284 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:28.849241    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aa9221-78f45b9f77-tgbmj"]
Nov 03 13:03:29.608064 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:29.608022    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:29.681760 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:29.681728    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" path="/var/lib/kubelet/pods/bcf93bee-986a-41ed-82a9-8230a01d3ef9/volumes"
Nov 03 13:03:29.793023 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:29.792986    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:31.270353 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:31.270320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:31.452604 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:31.452570    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:32.769719 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:32.769673    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:33.243550 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:33.243514    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:34.216049 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:34.216017    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:34.386593 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:34.386558    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:34.745694 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:34.745659    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:38.768570 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:38.768533    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:40.408551 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:40.408516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:40.572489 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:40.572453    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:43.071946 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:43.071866    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:43.241987 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:43.241954    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:43.398677 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:43.398641    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:47.722276 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:47.722226    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:52.155794 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:52.155757    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:53.604949 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:53.604914    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:53.767607 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:53.767566    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:54.622494 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:54.622455    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:03:56.472556 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:03:56.472514    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:03:56.641060 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:03:56.641022    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:01.135345 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:01.135302    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:04.244838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:04.244800    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:05.614652 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:05.614613    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:09.989117 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:09.989084    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:14.417863 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:14.417828    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:18.792160 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:18.792118    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:23.203274 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:23.203242    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:26.474450 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:04:26.474411    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:04:28.047176 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:28.047146    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:28.895491 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:28.895456    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:29.304202 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:29.304122    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:29.866364 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:29.866319    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:30.347233 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:30.347203    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:32.432214 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:32.432177    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:34.153085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:34.153052    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:34.324287 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:34.324243    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:34.495146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:34.495060    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:37.165569 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:37.165529    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:37.327204 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:37.327175    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:37.483224 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:37.483138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:41.962971 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:41.962933    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:46.351220 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:46.351181    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:47.433525 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:47.433487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:49.980306 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:49.980254    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:50.817773 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:50.817737    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:53.090873 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:53.090842    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:55.285981 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:55.285933    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:55.858800 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:55.858760    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:04:56.471596 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:04:56.471537    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:04:59.634910 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:04:59.634876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:04.028801 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:04.028770    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:04.552546 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:04.552511    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:08.636353 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:08.636314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:12.832898 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:12.832799    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:13.228553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:13.228519    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:13.660258 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:13.660229    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:13.819823 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:13.819784    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:14.204724 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:14.204677    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:17.723346 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:17.723306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:20.085760 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:20.085727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:21.721198 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:21.721157    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:22.209189 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:22.209157    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:26.473516 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:05:26.473472    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:05:26.714371 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:26.714331    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:31.211545 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:31.211509    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:35.606649 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:35.606609    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:40.013690 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:40.013652    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:41.378734 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:41.378700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:44.430441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:44.430364    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:48.888600 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:48.888565    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:52.977353 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.977319    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:05:52.978011 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.977992    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3066862a-c259-4bac-96a9-9da266214166" containerName="payload-processing"
Nov 03 13:05:52.978078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.978013    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3066862a-c259-4bac-96a9-9da266214166" containerName="payload-processing"
Nov 03 13:05:52.978078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.978030    5784 cpu_manager.go:414] "containerMap: removing container" podUID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" containerName="maas-api"
Nov 03 13:05:52.978078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.978035    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" containerName="maas-api"
Nov 03 13:05:52.978194 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.978163    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3066862a-c259-4bac-96a9-9da266214166" containerName="payload-processing"
Nov 03 13:05:52.978194 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.978173    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcf93bee-986a-41ed-82a9-8230a01d3ef9" containerName="maas-api"
Nov 03 13:05:52.981827 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:52.981807    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.030576 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.030547    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5e9ee9-serving-cert\""
Nov 03 13:05:53.079842 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.079807    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:05:53.127183 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.127149    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrbvz\" (UniqueName: \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.127399 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.127270    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.127563 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.127539    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.127643 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.127619    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.201093 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.201058    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:53.229100 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.229022    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.229100 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.229055    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.229321 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.229159    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrbvz\" (UniqueName: \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.229321 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.229209    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.229976 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.229950    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.231560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.231533    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.231693 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.231666    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.281837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.281806    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrbvz\" (UniqueName: \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\") pod \"maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.295048 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.295025    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:53.345814 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.345778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:05:53.669724 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.669695    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:05:53.670546 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:05:53.670522    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafab2659_0852_4586_afb4_770b89cef52a.slice/crio-acc6ac9ab2ddfd48d81c1a649dd11279f015d0c95733e9cd3f16e6fb8b5344eb WatchSource:0}: Error finding container acc6ac9ab2ddfd48d81c1a649dd11279f015d0c95733e9cd3f16e6fb8b5344eb: Status 404 returned error can't find the container with id acc6ac9ab2ddfd48d81c1a649dd11279f015d0c95733e9cd3f16e6fb8b5344eb
Nov 03 13:05:53.671930 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.671913    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:05:53.718600 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:53.718567    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" event={"ID":"afab2659-0852-4586-afb4-770b89cef52a","Type":"ContainerStarted","Data":"acc6ac9ab2ddfd48d81c1a649dd11279f015d0c95733e9cd3f16e6fb8b5344eb"}
Nov 03 13:05:55.739789 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:55.739745    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" event={"ID":"afab2659-0852-4586-afb4-770b89cef52a","Type":"ContainerStarted","Data":"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"}
Nov 03 13:05:55.740185 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:55.739830    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:05:55.853698 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:55.853642    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" podStartSLOduration=2.821940373 podStartE2EDuration="3.853626383s" totalImagesPullingTime="1.03168601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:55.849062155 +0000 UTC m=+4462.291264457" watchObservedRunningTime="2026-11-03 13:05:55.853626383 +0000 UTC m=+4462.295828688"
Nov 03 13:05:56.472258 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:05:56.472223    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:05:57.813178 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:05:57.813139    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:01.753576 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:01.753547    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:06:02.291528 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:02.291492    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:04.869471 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:04.869437    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:06.674045 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:06.673993    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:11.053257 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:11.053223    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:15.549269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:15.549198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:17.506658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:17.506627    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:17.675822 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:17.675773    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:18.023044 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:18.023005    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:18.193493 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:18.193460    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:19.415463 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:19.415276    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:19.582476 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:19.582437    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:20.054253 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:20.054214    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:20.530205 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:20.530169    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:20.697528 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:20.697478    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:21.306539 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:21.306509    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:24.544439 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:24.544412    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:26.471533 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:06:26.471492    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:06:28.920394 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:28.920335    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:30.659128 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:30.659089    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:32.370930 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:32.370890    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:33.340528 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:33.340492    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:34.496845 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:34.496819    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:06:34.540295 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:34.540270    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:06:37.832426 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:37.831950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:38.200425 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:38.200178    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:39.211575 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:39.211535    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-dspjl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:06:39.212195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:39.212161    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:06:42.443931 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:42.443843    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:44.957948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:44.957911    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:46.859679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:46.859642    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:47.387911 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:47.387879    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:50.829756 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:50.829719    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:51.252713 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:51.252674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:51.637557 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:51.637526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:51.787556 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:51.787521    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:53.547309 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:53.547272    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:56.024188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:56.024156    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:56.309658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:06:56.309577    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:06:56.472354 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:06:56.472316    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:07:00.458069 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:00.458023    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:04.771817 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:04.771778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:05.211460 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:05.211415    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:05.366608 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:05.366572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:09.247596 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:09.247538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:13.758071 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:13.758031    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:18.195981 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:18.195946    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:22.701000 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:22.700964    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:26.473353 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:07:26.473318    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:07:27.356496 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:27.356448    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:27.819113 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:27.819076    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:31.247976 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:31.247943    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:31.902812 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:31.902774    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:34.308677 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.308638    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:07:34.309176 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.308903    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery" containerID="cri-o://220da0bc9f0543b318489ddd97c9d963165282d8cb8ff768230696b3385a688c" gracePeriod=30
Nov 03 13:07:34.537554 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.537511    5784 generic.go:356] "Generic (PLEG): container finished" podID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerID="220da0bc9f0543b318489ddd97c9d963165282d8cb8ff768230696b3385a688c" exitCode=0
Nov 03 13:07:34.537775 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.537754    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" event={"ID":"3eb373aa-4f44-4513-895c-1bbd96f1ccb4","Type":"ContainerDied","Data":"220da0bc9f0543b318489ddd97c9d963165282d8cb8ff768230696b3385a688c"}
Nov 03 13:07:34.577741 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.577683    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:07:34.736959 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.736929    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737158 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.736986    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737158 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737084    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737158 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737127    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737357 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737189    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737357 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737265    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737557 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737413    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\") pod \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\" (UID: \"3eb373aa-4f44-4513-895c-1bbd96f1ccb4\") "
Nov 03 13:07:34.737738 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737715    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:07:34.738018 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.737997    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-ca-configmap\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.739402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.739348    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:07:34.739402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.739360    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:07:34.739648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.739621    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:07:34.739970 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.739939    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "kube-api-access-jq8kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:07:34.740065 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.739979    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:07:34.740065 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.740024    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig" pod "3eb373aa-4f44-4513-895c-1bbd96f1ccb4" (UID: "3eb373aa-4f44-4513-895c-1bbd96f1ccb4"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839602    5784 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-csr-dns-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839632    5784 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-cacerts\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839644    5784 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-local-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839653    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8kf\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-kube-api-access-jq8kf\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839662    5784 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-kubeconfig\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:34.839679 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:34.839671    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb373aa-4f44-4513-895c-1bbd96f1ccb4-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:07:35.546783 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:35.546751    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"
Nov 03 13:07:35.547279 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:35.546785    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2" event={"ID":"3eb373aa-4f44-4513-895c-1bbd96f1ccb4","Type":"ContainerDied","Data":"5e2fdfb522f850714f2f386b43a1c77d62eea77efb5c13c31fcce18a772197a6"}
Nov 03 13:07:35.547279 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:35.546850    5784 scope.go:122] "RemoveContainer" containerID="220da0bc9f0543b318489ddd97c9d963165282d8cb8ff768230696b3385a688c"
Nov 03 13:07:35.746818 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:35.746787    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:07:35.792892 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:35.792861    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-c5qn2"]
Nov 03 13:07:36.329924 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:36.329887    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:37.681541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:37.681507    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" path="/var/lib/kubelet/pods/3eb373aa-4f44-4513-895c-1bbd96f1ccb4/volumes"
Nov 03 13:07:40.710607 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:40.710556    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:45.197893 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:45.197856    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:49.590973 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:49.590927    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:53.514659 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:53.514627    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:55.125528 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:55.125478    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:07:56.471893 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:07:56.471851    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:07:59.524526 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:07:59.524490    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:02.635438 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:02.635403    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:02.788490 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:02.788439    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:04.117903 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:04.117841    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:04.279726 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:04.279695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:04.709331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:04.709288    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:05.138112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:05.138079    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:05.313332 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:05.313281    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:05.533742 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:05.533620    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:05.685143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:05.685092    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:08.924797 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:08.924762    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:11.275564 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:11.275469    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:11.434414 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:11.434366    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:13.323968 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:13.323934    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:13.859371 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:13.859333    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:16.373438 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:16.373400    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 13:08:16.537038 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:16.537009    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:17.751480 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:17.751447    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:20.149867 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:20.149822    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:22.269716 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:22.269672    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:22.560394 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:22.560295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:26.474442 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:08:26.474402    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:08:26.552441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.552406    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:08:26.553306 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.553285    5784 cpu_manager.go:414] "containerMap: removing container" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery"
Nov 03 13:08:26.553359 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.553310    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery"
Nov 03 13:08:26.553513 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.553501    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eb373aa-4f44-4513-895c-1bbd96f1ccb4" containerName="discovery"
Nov 03 13:08:26.557665 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.557644    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.608227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.608139    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5e9ee9\"/\"kube-root-ca.crt\""
Nov 03 13:08:26.609539 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.609514    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:08:26.629002 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.628975    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5e9ee9\"/\"openshift-service-ca.crt\""
Nov 03 13:08:26.646798 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.646759    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7bg\" (UniqueName: \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.646950 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.646807    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.647019 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.646987    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.647081 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.647047    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.647164 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.647137    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.647262 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.647239    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.649937 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.649915    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5e9ee9\"/\"default-dockercfg-l5gl8\""
Nov 03 13:08:26.650096 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.649931    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5e9ee9\"/\"test-model-5e9ee9-kserve-self-signed-certs\""
Nov 03 13:08:26.653242 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.653218    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:26.748465 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748341    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.748465 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748431    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.748721 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748497    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.748721 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748636    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7bg\" (UniqueName: \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.748721 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748668    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.748888 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.748789    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.749186 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.749160    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.750572 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.750531    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.750778 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.750757    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.756703 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.756670    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.761443 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.761365    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.801294 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.801256    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:26.802748 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.802719    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7bg\" (UniqueName: \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\") pod \"test-model-5e9ee9-kserve-599c9dd7f-nthcn\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") " pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:26.870824 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:26.870758    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:27.251760 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:27.251726    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:08:27.252729 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:08:27.252690    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb14c6130_c7ad_47c4_83dd_fa14a01dec35.slice/crio-9d02d99b0e52fabf11732ce3d33c809eda19e9b4afdd81d71c55a4234d03f7a2 WatchSource:0}: Error finding container 9d02d99b0e52fabf11732ce3d33c809eda19e9b4afdd81d71c55a4234d03f7a2: Status 404 returned error can't find the container with id 9d02d99b0e52fabf11732ce3d33c809eda19e9b4afdd81d71c55a4234d03f7a2
Nov 03 13:08:27.933998 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:27.933961    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" event={"ID":"b14c6130-c7ad-47c4-83dd-fa14a01dec35","Type":"ContainerStarted","Data":"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"}
Nov 03 13:08:27.933998 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:27.934004    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" event={"ID":"b14c6130-c7ad-47c4-83dd-fa14a01dec35","Type":"ContainerStarted","Data":"9d02d99b0e52fabf11732ce3d33c809eda19e9b4afdd81d71c55a4234d03f7a2"}
Nov 03 13:08:28.044204 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:28.044145    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" podStartSLOduration=2.044125358 podStartE2EDuration="2.044125358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:28.041658298 +0000 UTC m=+4614.483860622" watchObservedRunningTime="2026-11-03 13:08:28.044125358 +0000 UTC m=+4614.486327665"
Nov 03 13:08:28.871731 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:28.871694    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:28.898772 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:28.898717    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:29.061060 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:29.061025    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:29.222400 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:29.222245    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:31.394009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.393973    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" containerID="cri-o://0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9" gracePeriod=45
Nov 03 13:08:31.521743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.521709    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.71:8000/ready\": dial tcp 10.134.0.71:8000: connect: connection refused"
Nov 03 13:08:31.573961 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.573818    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:31.661816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.661746    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 13:08:31.739191 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739158    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739413 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739208    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739413 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739252    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739413 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739288    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739614 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739412    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\" (UniqueName: \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739614 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739479    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\") pod \"6ab07899-0852-49b6-807b-84a24a08f5c9\" (UID: \"6ab07899-0852-49b6-807b-84a24a08f5c9\") "
Nov 03 13:08:31.739614 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739595    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.739614 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739604    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.739818 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739766    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.740027 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.739996    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.740027 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.740022    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.740205 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.740037    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.741791 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.741769    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:08:31.741972 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.741952    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:08:31.742022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.741972    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz" pod "6ab07899-0852-49b6-807b-84a24a08f5c9" (UID: "6ab07899-0852-49b6-807b-84a24a08f5c9"). InnerVolumeSpecName "kube-api-access-9nfrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:08:31.842095 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.841763    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab07899-0852-49b6-807b-84a24a08f5c9-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.842095 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.841793    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab07899-0852-49b6-807b-84a24a08f5c9-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:31.842095 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:31.841804    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfrz\" (UniqueName: \"kubernetes.io/projected/6ab07899-0852-49b6-807b-84a24a08f5c9-kube-api-access-9nfrz\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:08:32.013912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.013801    5784 generic.go:356] "Generic (PLEG): container finished" podID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerID="0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9" exitCode=0
Nov 03 13:08:32.013912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.013875    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"
Nov 03 13:08:32.014169 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.014133    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" event={"ID":"6ab07899-0852-49b6-807b-84a24a08f5c9","Type":"ContainerDied","Data":"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"}
Nov 03 13:08:32.014237 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.014180    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm" event={"ID":"6ab07899-0852-49b6-807b-84a24a08f5c9","Type":"ContainerDied","Data":"db25feba59fe1618f65b5e9c999e9d77985a1b8a55a5b9ac35113ecaca19a8b7"}
Nov 03 13:08:32.014237 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.014207    5784 scope.go:122] "RemoveContainer" containerID="0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"
Nov 03 13:08:32.044544 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.044520    5784 scope.go:122] "RemoveContainer" containerID="0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"
Nov 03 13:08:32.044967 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:08:32.044936    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9\": container with ID starting with 0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9 not found: ID does not exist" containerID="0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"
Nov 03 13:08:32.045131 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.044972    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9"} err="failed to get container status \"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9\": rpc error: code = NotFound desc = could not find container \"0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9\": container with ID starting with 0dc03b82365290d8bd68edcde7480201c1fa212aab5bc0c72f88f5e1671671e9 not found: ID does not exist"
Nov 03 13:08:32.169859 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.169828    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 13:08:32.172856 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:32.172827    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8d2c7c/premium-w4-8d2c7c-kserve-5f598c4b7c-bswzm"]
Nov 03 13:08:33.683943 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:33.683909    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" path="/var/lib/kubelet/pods/6ab07899-0852-49b6-807b-84a24a08f5c9/volumes"
Nov 03 13:08:35.094263 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:35.094238    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:36.026236 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.026201    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:08:36.027009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.026985    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main"
Nov 03 13:08:36.027009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.027008    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main"
Nov 03 13:08:36.027205 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.027114    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab07899-0852-49b6-807b-84a24a08f5c9" containerName="main"
Nov 03 13:08:36.032340 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.032318    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.084101 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.084070    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:08:36.086804 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.086778    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"default-dockercfg-2gjqq\""
Nov 03 13:08:36.115250 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.115219    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"sub-test-model-cacb84-kserve-self-signed-certs\""
Nov 03 13:08:36.115250 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.115229    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"openshift-service-ca.crt\""
Nov 03 13:08:36.120491 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.120464    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:36.125503 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.125482    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-cacb84\"/\"kube-root-ca.crt\""
Nov 03 13:08:36.192345 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192309    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.192536 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192462    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.192536 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192493    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.192621 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192570    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lz6m\" (UniqueName: \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.192717 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192669    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.192717 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.192708    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.276912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.276812    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:36.294188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294148    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294367 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294204    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294367 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294234    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lz6m\" (UniqueName: \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294367 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294326    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294574 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294367    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294574 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294460    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294794 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294767    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294794 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294781    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.294915 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.294892    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.296990 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.296964    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.297110 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.297008    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.347874 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.347835    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lz6m\" (UniqueName: \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\") pod \"sub-test-model-cacb84-kserve-6db75d5f5-d9jbm\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") " pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.348532 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.348505    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:36.744977 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:08:36.744932    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod284f9899_7554_43e3_bc3d_2dc69e714e14.slice/crio-88a455b4a1d2a34387616238b8ba2a3305eac1cd25f7ed14fbe3eb1ecd3ea68b WatchSource:0}: Error finding container 88a455b4a1d2a34387616238b8ba2a3305eac1cd25f7ed14fbe3eb1ecd3ea68b: Status 404 returned error can't find the container with id 88a455b4a1d2a34387616238b8ba2a3305eac1cd25f7ed14fbe3eb1ecd3ea68b
Nov 03 13:08:36.745656 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.745626    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:08:36.871246 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.871209    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:36.883837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:36.883805    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:37.062512 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:37.062408    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" event={"ID":"284f9899-7554-43e3-bc3d-2dc69e714e14","Type":"ContainerStarted","Data":"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"}
Nov 03 13:08:37.062512 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:37.062463    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" event={"ID":"284f9899-7554-43e3-bc3d-2dc69e714e14","Type":"ContainerStarted","Data":"88a455b4a1d2a34387616238b8ba2a3305eac1cd25f7ed14fbe3eb1ecd3ea68b"}
Nov 03 13:08:37.075513 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:37.075487    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:08:37.170159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:37.170107    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" podStartSLOduration=2.170093377 podStartE2EDuration="2.170093377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:08:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:08:37.167752579 +0000 UTC m=+4623.609954903" watchObservedRunningTime="2026-11-03 13:08:37.170093377 +0000 UTC m=+4623.612295722"
Nov 03 13:08:37.349538 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:37.349455    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:40.514150 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:40.514105    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:44.940181 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:44.940138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:46.348893 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:46.348854    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:46.362325 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:46.362300    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:47.142814 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:47.142783    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:08:49.331050 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:49.330996    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:49.495255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:49.495218    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:53.820543 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:53.818135    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:53.989838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:53.989801    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:54.144946 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:54.144912    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:56.453676 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:56.453639    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:56.472594 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:08:56.472552    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:08:56.610494 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:56.610464    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:08:58.411531 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:08:58.411491    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:00.408636 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:00.408598    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:02.151535 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.151498    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 13:09:02.151947 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.151706    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" podUID="eb844ff7-5348-4de3-aa50-82fdef7539f7" containerName="maas-api" containerID="cri-o://fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933" gracePeriod=30
Nov 03 13:09:02.277009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.276979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:02.462643 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.462614    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 13:09:02.597194 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.597143    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\") pod \"eb844ff7-5348-4de3-aa50-82fdef7539f7\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") "
Nov 03 13:09:02.597415 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.597243    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") pod \"eb844ff7-5348-4de3-aa50-82fdef7539f7\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") "
Nov 03 13:09:02.597415 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.597278    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\") pod \"eb844ff7-5348-4de3-aa50-82fdef7539f7\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") "
Nov 03 13:09:02.597415 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.597330    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\" (UniqueName: \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\") pod \"eb844ff7-5348-4de3-aa50-82fdef7539f7\" (UID: \"eb844ff7-5348-4de3-aa50-82fdef7539f7\") "
Nov 03 13:09:02.598435 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.598369    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca" pod "eb844ff7-5348-4de3-aa50-82fdef7539f7" (UID: "eb844ff7-5348-4de3-aa50-82fdef7539f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:09:02.600869 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.600841    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb" pod "eb844ff7-5348-4de3-aa50-82fdef7539f7" (UID: "eb844ff7-5348-4de3-aa50-82fdef7539f7"). InnerVolumeSpecName "kube-api-access-pt9xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:09:02.601533 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.601485    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs" pod "eb844ff7-5348-4de3-aa50-82fdef7539f7" (UID: "eb844ff7-5348-4de3-aa50-82fdef7539f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:02.620613 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.620572    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls" pod "eb844ff7-5348-4de3-aa50-82fdef7539f7" (UID: "eb844ff7-5348-4de3-aa50-82fdef7539f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:09:02.698852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.698760    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:02.698852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.698788    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb844ff7-5348-4de3-aa50-82fdef7539f7-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:02.698852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.698800    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt9xb\" (UniqueName: \"kubernetes.io/projected/eb844ff7-5348-4de3-aa50-82fdef7539f7-kube-api-access-pt9xb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:02.698852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:02.698814    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb844ff7-5348-4de3-aa50-82fdef7539f7-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:09:03.245633 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.245592    5784 generic.go:356] "Generic (PLEG): container finished" podID="eb844ff7-5348-4de3-aa50-82fdef7539f7" containerID="fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933" exitCode=0
Nov 03 13:09:03.245633 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.245634    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"
Nov 03 13:09:03.246199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.245673    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" event={"ID":"eb844ff7-5348-4de3-aa50-82fdef7539f7","Type":"ContainerDied","Data":"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"}
Nov 03 13:09:03.246199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.245737    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6" event={"ID":"eb844ff7-5348-4de3-aa50-82fdef7539f7","Type":"ContainerDied","Data":"e07df703af2ce4b677cbbea88ad7b28a2b7777fe30b0c09054e734e96833c075"}
Nov 03 13:09:03.246199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.245770    5784 scope.go:122] "RemoveContainer" containerID="fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"
Nov 03 13:09:03.259085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.259059    5784 scope.go:122] "RemoveContainer" containerID="fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"
Nov 03 13:09:03.259522 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:09:03.259498    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933\": container with ID starting with fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933 not found: ID does not exist" containerID="fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"
Nov 03 13:09:03.259597 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.259531    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933"} err="failed to get container status \"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933\": rpc error: code = NotFound desc = could not find container \"fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933\": container with ID starting with fb705dd545fe6e4c29dc300d0cf828e10c8cc02370800593184984edbc06c933 not found: ID does not exist"
Nov 03 13:09:03.399022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.398986    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 13:09:03.404315 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.404283    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-8d2c7c-6bf9c74698-m6rm6"]
Nov 03 13:09:03.686405 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.686261    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb844ff7-5348-4de3-aa50-82fdef7539f7" path="/var/lib/kubelet/pods/eb844ff7-5348-4de3-aa50-82fdef7539f7/volumes"
Nov 03 13:09:03.859267 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:03.859233    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:04.124755 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:04.124668    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.281600 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:05.281567    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.747806 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:05.747755    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:05.901405 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:05.900845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:08.673121 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:08.673089    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:13.141942 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:13.141906    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:17.610595 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:17.610552    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:22.117589 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:22.117550    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:23.667751 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:23.667695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:26.475418 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:09:26.475369    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:09:26.496899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:26.496870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:29.577091 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:29.577057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:29.738499 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:29.738468    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:30.885673 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:30.885625    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:35.263790 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:35.263740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:39.756677 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:39.756640    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:44.213509 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:44.213305    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:48.596698 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:48.596630    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:52.967694 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:52.967660    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:09:56.471950 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:09:56.471904    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:09:57.435082 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:09:57.435047    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:01.820904 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:01.820871    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:06.159149 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:06.159115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:06.307305 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:06.307266    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:10.646139 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:10.646100    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:15.046143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:15.046099    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:16.946670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:16.946629    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:19.510159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:19.510125    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:23.591507 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.591461    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:10:23.592087 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.592071    5784 cpu_manager.go:414] "containerMap: removing container" podUID="eb844ff7-5348-4de3-aa50-82fdef7539f7" containerName="maas-api"
Nov 03 13:10:23.592087 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.592086    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb844ff7-5348-4de3-aa50-82fdef7539f7" containerName="maas-api"
Nov 03 13:10:23.592221 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.592181    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb844ff7-5348-4de3-aa50-82fdef7539f7" containerName="maas-api"
Nov 03 13:10:23.596031 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.596006    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.652009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.651964    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:10:23.686497 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686458    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.686691 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686518    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.686691 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686539    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2jx\" (UniqueName: \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.686691 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686657    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.686858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686733    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.686858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.686810    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.690909 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.690870    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"kube-root-ca.crt\""
Nov 03 13:10:23.691036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.690919    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"sub-test-model-e0615c-kserve-self-signed-certs\""
Nov 03 13:10:23.694537 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.694520    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"default-dockercfg-htfq5\""
Nov 03 13:10:23.694618 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.694529    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e0615c\"/\"openshift-service-ca.crt\""
Nov 03 13:10:23.705931 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.705909    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:23.787712 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787680    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.787899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787776    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.787899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787833    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.787899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787848    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2jx\" (UniqueName: \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.787899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787872    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.788142 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.787906    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.788225 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.788206    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.788575 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.788539    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.788575 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.788562    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.790199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.790171    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.790319 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.790296    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.842217 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.842136    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2jx\" (UniqueName: \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\") pod \"sub-test-model-e0615c-kserve-58cbf45f45-692kr\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") " pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:23.908029 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:23.907994    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:24.297166 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:24.297136    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:10:24.298012 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:10:24.297975    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20968277_ad9a_46f5_bc9a_d6384af5fb02.slice/crio-ca65dfb55808a47e789a9f0b575377f80736623fa8882429adbc29fc4f457988 WatchSource:0}: Error finding container ca65dfb55808a47e789a9f0b575377f80736623fa8882429adbc29fc4f457988: Status 404 returned error can't find the container with id ca65dfb55808a47e789a9f0b575377f80736623fa8882429adbc29fc4f457988
Nov 03 13:10:24.867418 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:24.867354    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" event={"ID":"20968277-ad9a-46f5-bc9a-d6384af5fb02","Type":"ContainerStarted","Data":"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"}
Nov 03 13:10:24.867418 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:24.867423    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" event={"ID":"20968277-ad9a-46f5-bc9a-d6384af5fb02","Type":"ContainerStarted","Data":"ca65dfb55808a47e789a9f0b575377f80736623fa8882429adbc29fc4f457988"}
Nov 03 13:10:24.909024 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:24.908986    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:24.973455 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:24.973373    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" podStartSLOduration=1.973359929 podStartE2EDuration="1.973359929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:24.972405329 +0000 UTC m=+4731.414607651" watchObservedRunningTime="2026-11-03 13:10:24.973359929 +0000 UTC m=+4731.415562239"
Nov 03 13:10:26.035222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:26.035159    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:26.474247 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:10:26.474206    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:10:28.321513 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:28.321475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:32.726952 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:32.726884    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:33.908320 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:33.908286    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:33.921398 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:33.921355    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:33.949124 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:33.949092    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:10:36.249344 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:36.249306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:37.258183 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:37.258147    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:41.721177 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:41.721143    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:46.200792 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:46.200743    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:50.619559 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:50.619519    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:55.003354 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:55.003320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:10:56.473892 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:10:56.473849    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:10:59.402538 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:10:59.402496    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:03.835296 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:03.835257    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:05.117694 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:05.117650    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:06.490369 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:06.490329    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:08.314520 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:08.314485    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:12.705051 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:12.704961    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:17.127078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:17.127027    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:21.659969 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:21.659928    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:26.000738 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:26.000704    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:26.472750 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:11:26.472714    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:11:30.508538 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:30.508501    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:34.553195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:34.553165    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:11:34.597469 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:34.597445    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:11:34.929280 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:34.929243    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:36.699415 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:36.699367    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:39.429350 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:39.429316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:43.951605 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:43.951502    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:48.423576 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:48.423538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:52.946655 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:52.946616    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:55.145355 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:55.145320    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:11:56.474033 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:11:56.473984    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:11:57.352362 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:11:57.352326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:01.788957 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:01.788921    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:06.147598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:06.147554    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:06.779553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:06.779521    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:08.497931 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:08.497896    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:10.552119 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:10.552082    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:14.912394 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:14.912346    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:19.317557 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:19.317523    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:23.860884 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:23.860846    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:26.472084 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:12:26.472034    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:12:28.297940 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:28.297870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:32.741754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:32.741702    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:37.140764 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:37.140727    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:41.611319 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:41.611210    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:43.564514 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:43.564478    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:12:43.599236 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:43.599186    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:44.036882 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:44.036845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:46.033607 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:46.033573    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:50.116448 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:50.116408    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:50.566086 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:50.566040    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:50.716063 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:50.716027    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:50.873866 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:50.873750    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:55.131478 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:55.131430    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:55.609419 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:55.609326    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:56.474276 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:12:56.474239    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:12:58.585152 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.585100    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" podUID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" containerName="main" containerID="cri-o://2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6" gracePeriod=45
Nov 03 13:12:58.868244 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.868211    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:12:58.952039 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.952003    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.952827 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.952801    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.953070 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.953051    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.953886 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.953396    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:58.954000 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.953445    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:58.954090 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.953870    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\" (UniqueName: \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.954290 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.954270    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\" (UniqueName: \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.954429 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.954309    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\") pod \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\" (UID: \"b14c6130-c7ad-47c4-83dd-fa14a01dec35\") "
Nov 03 13:12:58.954747 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.954723    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:58.954833 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.954770    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:58.955014 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.954996    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:58.956831 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.956785    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:12:58.959436 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.959412    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:12:58.963864 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:58.963830    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg" pod "b14c6130-c7ad-47c4-83dd-fa14a01dec35" (UID: "b14c6130-c7ad-47c4-83dd-fa14a01dec35"). InnerVolumeSpecName "kube-api-access-2c7bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:12:59.004188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.004149    5784 generic.go:356] "Generic (PLEG): container finished" podID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" containerID="2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6" exitCode=0
Nov 03 13:12:59.004188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.004187    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"
Nov 03 13:12:59.004473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.004222    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" event={"ID":"b14c6130-c7ad-47c4-83dd-fa14a01dec35","Type":"ContainerDied","Data":"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"}
Nov 03 13:12:59.004473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.004264    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn" event={"ID":"b14c6130-c7ad-47c4-83dd-fa14a01dec35","Type":"ContainerDied","Data":"9d02d99b0e52fabf11732ce3d33c809eda19e9b4afdd81d71c55a4234d03f7a2"}
Nov 03 13:12:59.004473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.004280    5784 scope.go:122] "RemoveContainer" containerID="2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"
Nov 03 13:12:59.019852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.019831    5784 scope.go:122] "RemoveContainer" containerID="2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"
Nov 03 13:12:59.020200 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:12:59.020172    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6\": container with ID starting with 2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6 not found: ID does not exist" containerID="2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"
Nov 03 13:12:59.020295 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.020204    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6"} err="failed to get container status \"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6\": rpc error: code = NotFound desc = could not find container \"2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6\": container with ID starting with 2b0896e225de9f28be6db9bbd58c8f7e9417afe5ac9c4fcd4f173593c3c2cfb6 not found: ID does not exist"
Nov 03 13:12:59.055948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.055907    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:59.055948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.055939    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7bg\" (UniqueName: \"kubernetes.io/projected/b14c6130-c7ad-47c4-83dd-fa14a01dec35-kube-api-access-2c7bg\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:59.055948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.055953    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b14c6130-c7ad-47c4-83dd-fa14a01dec35-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:59.056196 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.055967    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b14c6130-c7ad-47c4-83dd-fa14a01dec35-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:12:59.157129 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.157094    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:12:59.160318 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.160290    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5e9ee9/test-model-5e9ee9-kserve-599c9dd7f-nthcn"]
Nov 03 13:12:59.576331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.576237    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:12:59.681899 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:12:59.681863    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" path="/var/lib/kubelet/pods/b14c6130-c7ad-47c4-83dd-fa14a01dec35/volumes"
Nov 03 13:13:03.979304 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:03.979271    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:07.108644 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:07.108612    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:07.278139 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:07.278104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:08.446808 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:08.446771    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:12.818120 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:12.818073    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:17.227299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:17.227262    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:21.782599 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:21.782565    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:26.114613 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:26.114580    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:26.472856 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:26.472815    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:13:31.891246 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:31.891215    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:32.053553 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:32.053514    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:35.308898 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:35.308868    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:35.472318 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:35.472241    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:35.960491 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:35.960450    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:36.819330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:36.819299    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:37.030608 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:37.030571    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:37.370926 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:37.370894    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:38.118412 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:38.118362    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:40.719571 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:40.719538    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:41.182154 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.182121    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-50fe59"]
Nov 03 13:13:41.182808 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.182783    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" containerName="main"
Nov 03 13:13:41.182808 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.182805    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" containerName="main"
Nov 03 13:13:41.183000 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.182925    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b14c6130-c7ad-47c4-83dd-fa14a01dec35" containerName="main"
Nov 03 13:13:41.187673 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.187654    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:41.237596 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.237557    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-50fe59"]
Nov 03 13:13:41.254838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.254803    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbklb\" (UniqueName: \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\") pod \"test-curl-7596-50fe59\" (UID: \"06438e1b-7376-43c1-8515-2fb2b5dfeac6\") " pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:41.279659 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.279619    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4ghw4\""
Nov 03 13:13:41.355887 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.355851    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbklb\" (UniqueName: \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\") pod \"test-curl-7596-50fe59\" (UID: \"06438e1b-7376-43c1-8515-2fb2b5dfeac6\") " pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:41.408252 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.408221    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbklb\" (UniqueName: \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\") pod \"test-curl-7596-50fe59\" (UID: \"06438e1b-7376-43c1-8515-2fb2b5dfeac6\") " pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:41.500307 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.500213    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:41.866571 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:41.866542    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-50fe59"]
Nov 03 13:13:41.869152 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:13:41.869125    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06438e1b_7376_43c1_8515_2fb2b5dfeac6.slice/crio-92ad1e5853a326e40fba149674f8386c97d40fa70eb77f04bdff4ea83c738a9c WatchSource:0}: Error finding container 92ad1e5853a326e40fba149674f8386c97d40fa70eb77f04bdff4ea83c738a9c: Status 404 returned error can't find the container with id 92ad1e5853a326e40fba149674f8386c97d40fa70eb77f04bdff4ea83c738a9c
Nov 03 13:13:42.320706 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:42.320663    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-50fe59" event={"ID":"06438e1b-7376-43c1-8515-2fb2b5dfeac6","Type":"ContainerStarted","Data":"9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4"}
Nov 03 13:13:42.320706 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:42.320709    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-50fe59" event={"ID":"06438e1b-7376-43c1-8515-2fb2b5dfeac6","Type":"ContainerStarted","Data":"92ad1e5853a326e40fba149674f8386c97d40fa70eb77f04bdff4ea83c738a9c"}
Nov 03 13:13:42.428322 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:42.428267    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7596-50fe59" podStartSLOduration=1.428251887 podStartE2EDuration="1.428251887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:42.428081446 +0000 UTC m=+4928.870283751" watchObservedRunningTime="2026-11-03 13:13:42.428251887 +0000 UTC m=+4928.870454192"
Nov 03 13:13:43.851834 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.851793    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-50fe59"]
Nov 03 13:13:43.857133 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.856621    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-50fe59"]
Nov 03 13:13:43.974649 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.974610    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-8b12fc"]
Nov 03 13:13:43.975345 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.975324    5784 cpu_manager.go:414] "containerMap: removing container" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" containerName="test-curl-7596-50fe59"
Nov 03 13:13:43.975345 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.975343    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" containerName="test-curl-7596-50fe59"
Nov 03 13:13:43.975541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.975463    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" containerName="test-curl-7596-50fe59"
Nov 03 13:13:43.978877 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:43.978858    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:44.028071 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.028032    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-8b12fc"]
Nov 03 13:13:44.087866 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.087830    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbxw4\" (UniqueName: \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\") pod \"test-curl-7596-8b12fc\" (UID: \"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8\") " pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:44.189096 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.189052    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbxw4\" (UniqueName: \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\") pod \"test-curl-7596-8b12fc\" (UID: \"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8\") " pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:44.242484 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.242454    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbxw4\" (UniqueName: \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\") pod \"test-curl-7596-8b12fc\" (UID: \"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8\") " pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:44.291714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.291682    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:44.334792 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.334734    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7596-50fe59" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" containerName="test-curl-7596-50fe59" containerID="cri-o://9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4" gracePeriod=2
Nov 03 13:13:44.382048 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:44.382005    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-50fe59\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:44.461671 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.461582    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:44.664189 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.664162    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-8b12fc"]
Nov 03 13:13:44.665697 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:13:44.665662    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd018a35a_eb4e_4e16_9c15_4bb3ed19cae8.slice/crio-0a2bf167f7f98cfb7a6347cb19087b2327ead14c32b5c5ed8a1f0d78b9267414 WatchSource:0}: Error finding container 0a2bf167f7f98cfb7a6347cb19087b2327ead14c32b5c5ed8a1f0d78b9267414: Status 404 returned error can't find the container with id 0a2bf167f7f98cfb7a6347cb19087b2327ead14c32b5c5ed8a1f0d78b9267414
Nov 03 13:13:44.966047 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:44.965996    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:45.117754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:45.117654    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:45.297576 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:45.297526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:45.344045 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:45.344001    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-8b12fc" event={"ID":"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8","Type":"ContainerStarted","Data":"fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4"}
Nov 03 13:13:45.344230 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:45.344058    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-8b12fc" event={"ID":"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8","Type":"ContainerStarted","Data":"0a2bf167f7f98cfb7a6347cb19087b2327ead14c32b5c5ed8a1f0d78b9267414"}
Nov 03 13:13:45.449942 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:45.449863    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7596-8b12fc" podStartSLOduration=2.449845412 podStartE2EDuration="2.449845412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:13:45.446530048 +0000 UTC m=+4931.888732355" watchObservedRunningTime="2026-11-03 13:13:45.449845412 +0000 UTC m=+4931.892047717"
Nov 03 13:13:45.493597 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:45.493562    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-50fe59\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:46.545192 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.545158    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:46.621963 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.621884    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\" (UniqueName: \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\") pod \"06438e1b-7376-43c1-8515-2fb2b5dfeac6\" (UID: \"06438e1b-7376-43c1-8515-2fb2b5dfeac6\") "
Nov 03 13:13:46.624000 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.623979    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb" pod "06438e1b-7376-43c1-8515-2fb2b5dfeac6" (UID: "06438e1b-7376-43c1-8515-2fb2b5dfeac6"). InnerVolumeSpecName "kube-api-access-fbklb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:46.723786 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.723751    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbklb\" (UniqueName: \"kubernetes.io/projected/06438e1b-7376-43c1-8515-2fb2b5dfeac6-kube-api-access-fbklb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:46.802599 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.802558    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-8b12fc"]
Nov 03 13:13:46.806823 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:46.806596    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-8b12fc"]
Nov 03 13:13:47.363623 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.363587    5784 generic.go:356] "Generic (PLEG): container finished" podID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" containerID="9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4" exitCode=137
Nov 03 13:13:47.363838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.363645    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:47.363838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.363690    5784 scope.go:122] "RemoveContainer" containerID="9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4"
Nov 03 13:13:47.364001 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.363951    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7596-8b12fc" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" containerName="test-curl-7596-8b12fc" containerID="cri-o://fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4" gracePeriod=2
Nov 03 13:13:47.373540 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.373514    5784 scope.go:122] "RemoveContainer" containerID="9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4"
Nov 03 13:13:47.373849 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:47.373826    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4\": container with ID starting with 9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4 not found: ID does not exist" containerID="9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4"
Nov 03 13:13:47.373929 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.373861    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4"} err="failed to get container status \"9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4\": rpc error: code = NotFound desc = could not find container \"9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4\": container with ID starting with 9616f93f6f5c46e5d5964ef92e10c3d85f2dc36608ac0ba61f5e9238f2d6d0d4 not found: ID does not exist"
Nov 03 13:13:47.411122 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:47.411082    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-8b12fc\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:47.458145 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:47.458100    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-8b12fc\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:47.505450 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:47.505411    5784 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-50fe59\" is forbidden: User \"system:node:ip-10-0-139-140.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-140.us-west-2.compute.internal' and this object" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" pod="opendatahub/test-curl-7596-50fe59"
Nov 03 13:13:47.682446 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:47.682412    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06438e1b-7376-43c1-8515-2fb2b5dfeac6" path="/var/lib/kubelet/pods/06438e1b-7376-43c1-8515-2fb2b5dfeac6/volumes"
Nov 03 13:13:48.806640 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:48.806608    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:48.977810 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:48.977778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:49.606336 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.606307    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:49.657089 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.657048    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\" (UniqueName: \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\") pod \"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8\" (UID: \"d018a35a-eb4e-4e16-9c15-4bb3ed19cae8\") "
Nov 03 13:13:49.659953 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.659897    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4" pod "d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" (UID: "d018a35a-eb4e-4e16-9c15-4bb3ed19cae8"). InnerVolumeSpecName "kube-api-access-fbxw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:13:49.689692 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.689658    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" path="/var/lib/kubelet/pods/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8/volumes"
Nov 03 13:13:49.759001 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.758966    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbxw4\" (UniqueName: \"kubernetes.io/projected/d018a35a-eb4e-4e16-9c15-4bb3ed19cae8-kube-api-access-fbxw4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:13:49.922986 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:49.922902    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:50.102232 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.102170    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:50.387272 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.387242    5784 generic.go:356] "Generic (PLEG): container finished" podID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" containerID="fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4" exitCode=137
Nov 03 13:13:50.387474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.387301    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-8b12fc"
Nov 03 13:13:50.387474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.387342    5784 scope.go:122] "RemoveContainer" containerID="fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4"
Nov 03 13:13:50.396733 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.396716    5784 scope.go:122] "RemoveContainer" containerID="fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4"
Nov 03 13:13:50.397046 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:50.397024    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4\": container with ID starting with fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4 not found: ID does not exist" containerID="fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4"
Nov 03 13:13:50.397124 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:50.397059    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4"} err="failed to get container status \"fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4\": rpc error: code = NotFound desc = could not find container \"fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4\": container with ID starting with fcdf6942883ce3054369eae4a44a7998036e7e125569894cced6aa2d3f96a3c4 not found: ID does not exist"
Nov 03 13:13:53.162481 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:53.162445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:54.304208 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:54.304174    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:56.473395 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:13:56.473349    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:13:58.696945 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:58.696913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:13:58.845804 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:58.845760    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 13:13:58.846036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:58.845989    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy" containerID="cri-o://aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00" gracePeriod=30
Nov 03 13:13:58.891802 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:58.891768    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 13:13:58.892003 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:13:58.891980    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy" containerID="cri-o://e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03" gracePeriod=30
Nov 03 13:14:03.139254 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:03.139219    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:04.190118 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.190092    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 13:14:04.194033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.194012    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 13:14:04.337738 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.337703    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.337935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.337747    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.337935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.337783    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.337935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.337826    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.338159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338129    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.338331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338160    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.338331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338167    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.338331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338239    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.338331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338287    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.338331 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338316    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.339115 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338344    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.339115 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338394    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.339115 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.338556    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.339115 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339082    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339149    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339184    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339256    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339301    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339540    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\") pod \"01468df6-1754-4cbe-b664-03e846b35b71\" (UID: \"01468df6-1754-4cbe-b664-03e846b35b71\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339627    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339682    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\") pod \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\" (UID: \"2952de83-4b48-4b89-ba3d-4d2f9deb4e32\") "
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339540    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339697    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.339876    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340013    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340204    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340391    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340616    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340619    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340694    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340713    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340751    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340767    5784 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01468df6-1754-4cbe-b664-03e846b35b71-istiod-ca-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340781    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340796    5784 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-credential-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340820    5784 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-workload-socket\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.340954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.340835    5784 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-workload-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.342494 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.342319    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.342494 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.342399    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.343662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.343609    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.343751 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.343724    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "kube-api-access-md9lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.343813 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.343803    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:14:04.343993 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.343930    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 13:14:04.344080 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.343993    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token" pod "2952de83-4b48-4b89-ba3d-4d2f9deb4e32" (UID: "2952de83-4b48-4b89-ba3d-4d2f9deb4e32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.344134 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.344075    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt" pod "01468df6-1754-4cbe-b664-03e846b35b71" (UID: "01468df6-1754-4cbe-b664-03e846b35b71"). InnerVolumeSpecName "kube-api-access-mhhpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.441984    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442014    5784 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-data\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442024    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01468df6-1754-4cbe-b664-03e846b35b71-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442037    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442050    5784 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-istio-token\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442059    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhhpt\" (UniqueName: \"kubernetes.io/projected/01468df6-1754-4cbe-b664-03e846b35b71-kube-api-access-mhhpt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442068 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442068    5784 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01468df6-1754-4cbe-b664-03e846b35b71-istio-podinfo\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442653 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442079    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md9lr\" (UniqueName: \"kubernetes.io/projected/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-kube-api-access-md9lr\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.442653 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.442087    5784 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2952de83-4b48-4b89-ba3d-4d2f9deb4e32-istio-envoy\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:04.498410 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.498349    5784 generic.go:356] "Generic (PLEG): container finished" podID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerID="aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00" exitCode=0
Nov 03 13:14:04.498598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.498438    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"
Nov 03 13:14:04.498598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.498438    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" event={"ID":"2952de83-4b48-4b89-ba3d-4d2f9deb4e32","Type":"ContainerDied","Data":"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"}
Nov 03 13:14:04.498598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.498554    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl" event={"ID":"2952de83-4b48-4b89-ba3d-4d2f9deb4e32","Type":"ContainerDied","Data":"aa282e00d6b5ec18576f79c9932508af9bab170e3a475bb2e53dd915f51e7b7f"}
Nov 03 13:14:04.498598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.498585    5784 scope.go:122] "RemoveContainer" containerID="aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"
Nov 03 13:14:04.500590 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.500565    5784 generic.go:356] "Generic (PLEG): container finished" podID="01468df6-1754-4cbe-b664-03e846b35b71" containerID="e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03" exitCode=0
Nov 03 13:14:04.500745 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.500601    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" event={"ID":"01468df6-1754-4cbe-b664-03e846b35b71","Type":"ContainerDied","Data":"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"}
Nov 03 13:14:04.500745 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.500629    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"
Nov 03 13:14:04.500745 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.500636    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m" event={"ID":"01468df6-1754-4cbe-b664-03e846b35b71","Type":"ContainerDied","Data":"7adc98162831d92c157408f89b88f771bb830c812c543e75daccfe286873c1f1"}
Nov 03 13:14:04.516082 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.516064    5784 scope.go:122] "RemoveContainer" containerID="aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"
Nov 03 13:14:04.516406 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:14:04.516365    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00\": container with ID starting with aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00 not found: ID does not exist" containerID="aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"
Nov 03 13:14:04.516492 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.516406    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00"} err="failed to get container status \"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00\": rpc error: code = NotFound desc = could not find container \"aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00\": container with ID starting with aabbeb99149c9fd5a845da7eed728b8e4bf336e95164d9dc7f00ba66e6e25a00 not found: ID does not exist"
Nov 03 13:14:04.516492 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.516427    5784 scope.go:122] "RemoveContainer" containerID="e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"
Nov 03 13:14:04.526405 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.526358    5784 scope.go:122] "RemoveContainer" containerID="e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"
Nov 03 13:14:04.526712 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:14:04.526691    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03\": container with ID starting with e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03 not found: ID does not exist" containerID="e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"
Nov 03 13:14:04.526862 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.526717    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03"} err="failed to get container status \"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03\": rpc error: code = NotFound desc = could not find container \"e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03\": container with ID starting with e91ab14078d5031b99d01814f239035b1f373a3eac431800bf1033c0095fef03 not found: ID does not exist"
Nov 03 13:14:04.654348 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.654316    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 13:14:04.659744 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.659217    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vw59m"]
Nov 03 13:14:04.759464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.759363    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 13:14:04.763729 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:04.763703    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-dspjl"]
Nov 03 13:14:05.685477 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:05.685432    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01468df6-1754-4cbe-b664-03e846b35b71" path="/var/lib/kubelet/pods/01468df6-1754-4cbe-b664-03e846b35b71/volumes"
Nov 03 13:14:05.686207 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:05.686183    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" path="/var/lib/kubelet/pods/2952de83-4b48-4b89-ba3d-4d2f9deb4e32/volumes"
Nov 03 13:14:07.410308 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:07.410274    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:07.624676 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:07.624646    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:12.047844 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:12.047757    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:16.564118 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:16.564080    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:19.734605 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:19.734565    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:21.087731 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:21.087689    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:21.967390 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:21.967344    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:25.403827 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:25.403791    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:26.472313 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:14:26.472279    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:14:29.813120 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:29.813073    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:34.228218 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:34.228180    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:38.645206 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:38.645166    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:43.066162 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:43.066130    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:47.447195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:47.447109    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:48.262853 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:48.262808    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:14:48.320775 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:48.320738    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:51.175697 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.175655    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:14:51.176349 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.175993    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" podUID="afab2659-0852-4586-afb4-770b89cef52a" containerName="maas-api" containerID="cri-o://479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4" gracePeriod=30
Nov 03 13:14:51.307552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.307517    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:51.493856 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.493524    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:14:51.586647 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.586613    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\" (UniqueName: \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\") pod \"afab2659-0852-4586-afb4-770b89cef52a\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") "
Nov 03 13:14:51.586863 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.586693    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\") pod \"afab2659-0852-4586-afb4-770b89cef52a\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") "
Nov 03 13:14:51.586863 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.586741    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\") pod \"afab2659-0852-4586-afb4-770b89cef52a\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") "
Nov 03 13:14:51.586994 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.586870    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\") pod \"afab2659-0852-4586-afb4-770b89cef52a\" (UID: \"afab2659-0852-4586-afb4-770b89cef52a\") "
Nov 03 13:14:51.587603 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.587570    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca" pod "afab2659-0852-4586-afb4-770b89cef52a" (UID: "afab2659-0852-4586-afb4-770b89cef52a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:14:51.589774 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.589727    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz" pod "afab2659-0852-4586-afb4-770b89cef52a" (UID: "afab2659-0852-4586-afb4-770b89cef52a"). InnerVolumeSpecName "kube-api-access-mrbvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:14:51.590021 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.589995    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs" pod "afab2659-0852-4586-afb4-770b89cef52a" (UID: "afab2659-0852-4586-afb4-770b89cef52a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:51.592808 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.592778    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls" pod "afab2659-0852-4586-afb4-770b89cef52a" (UID: "afab2659-0852-4586-afb4-770b89cef52a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:14:51.689020 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.688984    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:51.689197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.689103    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afab2659-0852-4586-afb4-770b89cef52a-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:51.689197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.689124    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afab2659-0852-4586-afb4-770b89cef52a-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:51.689197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.689140    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrbvz\" (UniqueName: \"kubernetes.io/projected/afab2659-0852-4586-afb4-770b89cef52a-kube-api-access-mrbvz\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:14:51.835420 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.835314    5784 generic.go:356] "Generic (PLEG): container finished" podID="afab2659-0852-4586-afb4-770b89cef52a" containerID="479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4" exitCode=0
Nov 03 13:14:51.835420 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.835370    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"
Nov 03 13:14:51.835616 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.835404    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" event={"ID":"afab2659-0852-4586-afb4-770b89cef52a","Type":"ContainerDied","Data":"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"}
Nov 03 13:14:51.835616 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.835470    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8" event={"ID":"afab2659-0852-4586-afb4-770b89cef52a","Type":"ContainerDied","Data":"acc6ac9ab2ddfd48d81c1a649dd11279f015d0c95733e9cd3f16e6fb8b5344eb"}
Nov 03 13:14:51.835616 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.835493    5784 scope.go:122] "RemoveContainer" containerID="479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"
Nov 03 13:14:51.847051 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.847023    5784 scope.go:122] "RemoveContainer" containerID="479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"
Nov 03 13:14:51.847332 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:14:51.847310    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4\": container with ID starting with 479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4 not found: ID does not exist" containerID="479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"
Nov 03 13:14:51.847427 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.847344    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4"} err="failed to get container status \"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4\": rpc error: code = NotFound desc = could not find container \"479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4\": container with ID starting with 479566cc26b02c4cc4016168fb23d4e8802a5d462753f487884a3abbf7984cf4 not found: ID does not exist"
Nov 03 13:14:51.988423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:51.988362    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:14:52.034319 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:52.034289    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e9ee9-7449d9fbf5-472w8"]
Nov 03 13:14:52.904936 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:52.904904    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:53.053597 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:53.053555    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:53.682026 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:53.681993    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afab2659-0852-4586-afb4-770b89cef52a" path="/var/lib/kubelet/pods/afab2659-0852-4586-afb4-770b89cef52a/volumes"
Nov 03 13:14:56.472692 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:14:56.472660    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:14:57.553391 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:57.553325    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:58.175100 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:58.175059    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:14:58.286627 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:58.286592    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:14:58.624026 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:14:58.623945    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:00.430590 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:00.430549    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:00.607862 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:00.607823    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:00.917443 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:00.917410    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:01.129003 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:01.128972    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:01.584928 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:01.584892    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:01.752008 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:01.751968    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:02.499660 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:02.499623    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:02.673604 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:02.673564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:02.938682 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:02.938651    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:03.290003 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.289912    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" podUID="284f9899-7554-43e3-bc3d-2dc69e714e14" containerName="main" containerID="cri-o://8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d" gracePeriod=45
Nov 03 13:15:03.559596 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.559538    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:15:03.625389 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625337    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\" (UniqueName: \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.625602 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625576    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.625658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625627    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.625858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625831    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.626006 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625863    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:03.626006 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625870    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\" (UniqueName: \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.626006 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.625951    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:03.626251 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.626008    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\") pod \"284f9899-7554-43e3-bc3d-2dc69e714e14\" (UID: \"284f9899-7554-43e3-bc3d-2dc69e714e14\") "
Nov 03 13:15:03.626310 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.626287    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:03.626606 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.626582    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.626714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.626610    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.626714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.626624    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.628159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.628137    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:03.628250 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.628204    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "kube-api-access-2lz6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:03.628300 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.628242    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm" pod "284f9899-7554-43e3-bc3d-2dc69e714e14" (UID: "284f9899-7554-43e3-bc3d-2dc69e714e14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:03.647655 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.647626    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:03.727300 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.727268    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/284f9899-7554-43e3-bc3d-2dc69e714e14-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.727300 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.727298    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/284f9899-7554-43e3-bc3d-2dc69e714e14-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.727738 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.727308    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lz6m\" (UniqueName: \"kubernetes.io/projected/284f9899-7554-43e3-bc3d-2dc69e714e14-kube-api-access-2lz6m\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:03.837246 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.837156    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:03.922858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.922823    5784 generic.go:356] "Generic (PLEG): container finished" podID="284f9899-7554-43e3-bc3d-2dc69e714e14" containerID="8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d" exitCode=0
Nov 03 13:15:03.923034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.922884    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"
Nov 03 13:15:03.923034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.922907    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" event={"ID":"284f9899-7554-43e3-bc3d-2dc69e714e14","Type":"ContainerDied","Data":"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"}
Nov 03 13:15:03.923034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.922949    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm" event={"ID":"284f9899-7554-43e3-bc3d-2dc69e714e14","Type":"ContainerDied","Data":"88a455b4a1d2a34387616238b8ba2a3305eac1cd25f7ed14fbe3eb1ecd3ea68b"}
Nov 03 13:15:03.923034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.922967    5784 scope.go:122] "RemoveContainer" containerID="8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"
Nov 03 13:15:03.932820 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.932799    5784 scope.go:122] "RemoveContainer" containerID="8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"
Nov 03 13:15:03.933086 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:15:03.933057    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d\": container with ID starting with 8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d not found: ID does not exist" containerID="8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"
Nov 03 13:15:03.933198 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:03.933096    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d"} err="failed to get container status \"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d\": rpc error: code = NotFound desc = could not find container \"8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d\": container with ID starting with 8a935a34c9664072902fd3db906a02d6d7e5e0f24ae00463f93c419c724d0b5d not found: ID does not exist"
Nov 03 13:15:04.031250 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:04.031202    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:04.080754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:04.080711    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:15:04.084915 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:04.084888    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-cacb84/sub-test-model-cacb84-kserve-6db75d5f5-d9jbm"]
Nov 03 13:15:05.681935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:05.681902    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284f9899-7554-43e3-bc3d-2dc69e714e14" path="/var/lib/kubelet/pods/284f9899-7554-43e3-bc3d-2dc69e714e14/volumes"
Nov 03 13:15:06.374700 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:06.374669    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:06.606317 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:06.606284    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:07.269562 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:07.269516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:07.761762 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:07.761725    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:08.110670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:08.110586    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:10.674510 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:10.674476    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:11.509188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:11.509152    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:11.690126 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:11.690094    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:11.968442 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:11.968375    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:12.157218 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:12.157174    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:13.195937 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.195901    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" podUID="20968277-ad9a-46f5-bc9a-d6384af5fb02" containerName="main" containerID="cri-o://69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066" gracePeriod=45
Nov 03 13:15:13.464606 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.464580    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:15:13.532143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532105    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532333 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532165    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\" (UniqueName: \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532333 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532247    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532333 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532302    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532568 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532367    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532568 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532446    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\" (UniqueName: \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\") pod \"20968277-ad9a-46f5-bc9a-d6384af5fb02\" (UID: \"20968277-ad9a-46f5-bc9a-d6384af5fb02\") "
Nov 03 13:15:13.532788 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532722    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:13.532788 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532782    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:13.532984 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.532817    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:13.533276 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.533254    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.533391 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.533302    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.533391 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.533316    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.534743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.534716    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "kube-api-access-mq2jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:13.535008 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.534988    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:13.535144 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.535002    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm" pod "20968277-ad9a-46f5-bc9a-d6384af5fb02" (UID: "20968277-ad9a-46f5-bc9a-d6384af5fb02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:15:13.634078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.634041    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20968277-ad9a-46f5-bc9a-d6384af5fb02-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.634078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.634072    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20968277-ad9a-46f5-bc9a-d6384af5fb02-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.634078 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.634084    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2jx\" (UniqueName: \"kubernetes.io/projected/20968277-ad9a-46f5-bc9a-d6384af5fb02-kube-api-access-mq2jx\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:13.992471 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.992436    5784 generic.go:356] "Generic (PLEG): container finished" podID="20968277-ad9a-46f5-bc9a-d6384af5fb02" containerID="69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066" exitCode=0
Nov 03 13:15:13.992678 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.992492    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"
Nov 03 13:15:13.992678 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.992496    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" event={"ID":"20968277-ad9a-46f5-bc9a-d6384af5fb02","Type":"ContainerDied","Data":"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"}
Nov 03 13:15:13.992678 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.992541    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr" event={"ID":"20968277-ad9a-46f5-bc9a-d6384af5fb02","Type":"ContainerDied","Data":"ca65dfb55808a47e789a9f0b575377f80736623fa8882429adbc29fc4f457988"}
Nov 03 13:15:13.992678 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:13.992557    5784 scope.go:122] "RemoveContainer" containerID="69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"
Nov 03 13:15:14.002567 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:14.002546    5784 scope.go:122] "RemoveContainer" containerID="69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"
Nov 03 13:15:14.002907 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:15:14.002889    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066\": container with ID starting with 69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066 not found: ID does not exist" containerID="69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"
Nov 03 13:15:14.002954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:14.002917    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066"} err="failed to get container status \"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066\": rpc error: code = NotFound desc = could not find container \"69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066\": container with ID starting with 69194e1020215c32244e29a71da8741423acfbd8cbf3ac3807a4a720c6a6b066 not found: ID does not exist"
Nov 03 13:15:14.149611 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:14.149570    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:15:14.183710 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:14.183677    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e0615c/sub-test-model-e0615c-kserve-58cbf45f45-692kr"]
Nov 03 13:15:15.682453 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:15.682416    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20968277-ad9a-46f5-bc9a-d6384af5fb02" path="/var/lib/kubelet/pods/20968277-ad9a-46f5-bc9a-d6384af5fb02/volumes"
Nov 03 13:15:16.860658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:16.860624    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:17.047132 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:17.047083    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:20.327164 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:20.327126    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:20.827633 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:20.827599    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 13:15:20.827927 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:20.827878    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" podUID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" containerName="maas-api" containerID="cri-o://31c0ea44be23fcb6b002a202726cb65ad84685af1c0ca2efe969cb711c25d495" gracePeriod=30
Nov 03 13:15:20.996896 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:20.996797    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:21.053902 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.053866    5784 generic.go:356] "Generic (PLEG): container finished" podID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" containerID="31c0ea44be23fcb6b002a202726cb65ad84685af1c0ca2efe969cb711c25d495" exitCode=0
Nov 03 13:15:21.054118 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.053920    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" event={"ID":"cc7fb1c3-dc45-416a-ae66-d14ef00867c9","Type":"ContainerDied","Data":"31c0ea44be23fcb6b002a202726cb65ad84685af1c0ca2efe969cb711c25d495"}
Nov 03 13:15:21.110769 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.110739    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 13:15:21.221222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.221190    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\") pod \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") "
Nov 03 13:15:21.221446 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.221239    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\") pod \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") "
Nov 03 13:15:21.221446 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.221422    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\" (UniqueName: \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\") pod \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") "
Nov 03 13:15:21.221565 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.221456    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") pod \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\" (UID: \"cc7fb1c3-dc45-416a-ae66-d14ef00867c9\") "
Nov 03 13:15:21.221772 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.221738    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca" pod "cc7fb1c3-dc45-416a-ae66-d14ef00867c9" (UID: "cc7fb1c3-dc45-416a-ae66-d14ef00867c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:21.222029 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.222006    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.223922 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.223873    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs" pod "cc7fb1c3-dc45-416a-ae66-d14ef00867c9" (UID: "cc7fb1c3-dc45-416a-ae66-d14ef00867c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:21.224112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.224089    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls" pod "cc7fb1c3-dc45-416a-ae66-d14ef00867c9" (UID: "cc7fb1c3-dc45-416a-ae66-d14ef00867c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:21.224199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.224163    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj" pod "cc7fb1c3-dc45-416a-ae66-d14ef00867c9" (UID: "cc7fb1c3-dc45-416a-ae66-d14ef00867c9"). InnerVolumeSpecName "kube-api-access-jzcbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:21.323269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.323240    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzcbj\" (UniqueName: \"kubernetes.io/projected/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-kube-api-access-jzcbj\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.323269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.323268    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:21.323470 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:21.323285    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7fb1c3-dc45-416a-ae66-d14ef00867c9-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:22.061955 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.061881    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"
Nov 03 13:15:22.062541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.061880    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c" event={"ID":"cc7fb1c3-dc45-416a-ae66-d14ef00867c9","Type":"ContainerDied","Data":"2eaf2d18c6dd4dc2a37c98a8d7c9a14f5503bb004d82ecee51a7f6a2441fbe7f"}
Nov 03 13:15:22.062541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.062023    5784 scope.go:122] "RemoveContainer" containerID="31c0ea44be23fcb6b002a202726cb65ad84685af1c0ca2efe969cb711c25d495"
Nov 03 13:15:22.216306 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.216272    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 13:15:22.262640 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.262606    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1250b3-64694c89dd-ss57c"]
Nov 03 13:15:22.663586 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:22.663541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:23.681349 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:23.681318    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" path="/var/lib/kubelet/pods/cc7fb1c3-dc45-416a-ae66-d14ef00867c9/volumes"
Nov 03 13:15:24.331343 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:24.331306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:24.524510 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:24.524470    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:25.822474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:25.822437    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:26.315775 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:26.315743    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:26.473516 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:15:26.473481    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:15:28.091295 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:28.091254    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:31.251080 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:31.251043    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:31.420218 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:31.420177    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:32.041861 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:32.041834    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:33.947989 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:33.947958    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:34.803339 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:34.803306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:34.961686 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:34.961652    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:36.880234 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:36.880189    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:39.854727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:39.854690    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:40.641841 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:40.641802    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 13:15:40.642070 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:40.642043    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" podUID="407e4d6e-0b50-4652-b230-13407245d514" containerName="maas-api" containerID="cri-o://4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b" gracePeriod=30
Nov 03 13:15:40.760736 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:40.760695    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:40.930769 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:40.930659    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 13:15:41.058927 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.058843    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\") pod \"407e4d6e-0b50-4652-b230-13407245d514\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") "
Nov 03 13:15:41.059102 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.058991    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\") pod \"407e4d6e-0b50-4652-b230-13407245d514\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") "
Nov 03 13:15:41.059102 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.059025    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\" (UniqueName: \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\") pod \"407e4d6e-0b50-4652-b230-13407245d514\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") "
Nov 03 13:15:41.059102 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.059088    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") pod \"407e4d6e-0b50-4652-b230-13407245d514\" (UID: \"407e4d6e-0b50-4652-b230-13407245d514\") "
Nov 03 13:15:41.059825 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.059778    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca" pod "407e4d6e-0b50-4652-b230-13407245d514" (UID: "407e4d6e-0b50-4652-b230-13407245d514"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:15:41.061599 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.061568    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls" pod "407e4d6e-0b50-4652-b230-13407245d514" (UID: "407e4d6e-0b50-4652-b230-13407245d514"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:41.061820 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.061789    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr" pod "407e4d6e-0b50-4652-b230-13407245d514" (UID: "407e4d6e-0b50-4652-b230-13407245d514"). InnerVolumeSpecName "kube-api-access-9kmlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:15:41.062013 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.061993    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs" pod "407e4d6e-0b50-4652-b230-13407245d514" (UID: "407e4d6e-0b50-4652-b230-13407245d514"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:15:41.161560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.161477    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:41.161560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.161516    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/407e4d6e-0b50-4652-b230-13407245d514-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:41.161560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.161526    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmlr\" (UniqueName: \"kubernetes.io/projected/407e4d6e-0b50-4652-b230-13407245d514-kube-api-access-9kmlr\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:41.161560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.161535    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/407e4d6e-0b50-4652-b230-13407245d514-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:15:41.193909 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.193874    5784 generic.go:356] "Generic (PLEG): container finished" podID="407e4d6e-0b50-4652-b230-13407245d514" containerID="4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b" exitCode=0
Nov 03 13:15:41.194085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.193921    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"
Nov 03 13:15:41.194085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.193923    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" event={"ID":"407e4d6e-0b50-4652-b230-13407245d514","Type":"ContainerDied","Data":"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"}
Nov 03 13:15:41.194085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.194037    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l" event={"ID":"407e4d6e-0b50-4652-b230-13407245d514","Type":"ContainerDied","Data":"78f4c9c6555ba9424658efdaca8ddcf718fb7458bb928399f1b177e576b1399e"}
Nov 03 13:15:41.194085 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.194067    5784 scope.go:122] "RemoveContainer" containerID="4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"
Nov 03 13:15:41.205002 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.204984    5784 scope.go:122] "RemoveContainer" containerID="4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"
Nov 03 13:15:41.205253 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:15:41.205237    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b\": container with ID starting with 4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b not found: ID does not exist" containerID="4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"
Nov 03 13:15:41.205304 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.205261    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b"} err="failed to get container status \"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b\": rpc error: code = NotFound desc = could not find container \"4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b\": container with ID starting with 4d2c3a930cf0000b9554605273137677d87250f3e8c040b46d621ed9f60ece9b not found: ID does not exist"
Nov 03 13:15:41.347414 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.347358    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 13:15:41.355530 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.355498    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-8570a0-757584d985-nbt7l"]
Nov 03 13:15:41.681875 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:41.681842    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407e4d6e-0b50-4652-b230-13407245d514" path="/var/lib/kubelet/pods/407e4d6e-0b50-4652-b230-13407245d514/volumes"
Nov 03 13:15:42.448567 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:42.448533    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:47.231947 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:47.231916    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:47.763552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:47.763516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:48.095411 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:48.095293    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:49.340065 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:49.340030    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:49.797159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:49.797126    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:50.242767 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:50.242732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:52.250769 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:52.250732    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:55.174748 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:55.174712    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 13:15:55.318587 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:55.318553    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:56.473879 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:15:56.473840    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:15:56.739769 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:56.739678    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:15:59.355356 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:15:59.355316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:00.605531 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:00.605498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:01.220420 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:01.220362    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:02.921477 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:02.921435    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:03.336175 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:03.336090    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:05.713515 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:05.713487    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:07.073227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:07.073198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:08.067748 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:08.067709    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:08.247736 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:08.247702    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:08.439271 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:08.439236    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:09.793109 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:09.793061    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:10.195848 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.195807    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main" containerID="cri-o://fe17dbcb1fb65c8bed158fa32f3a46b628fa86d418054f93c06bc3a6cf4f01d7" gracePeriod=45
Nov 03 13:16:10.401783 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.401736    5784 generic.go:356] "Generic (PLEG): container finished" podID="7919092e-f3e8-491a-987c-303bca4a3486" containerID="fe17dbcb1fb65c8bed158fa32f3a46b628fa86d418054f93c06bc3a6cf4f01d7" exitCode=0
Nov 03 13:16:10.401948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.401811    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" event={"ID":"7919092e-f3e8-491a-987c-303bca4a3486","Type":"ContainerDied","Data":"fe17dbcb1fb65c8bed158fa32f3a46b628fa86d418054f93c06bc3a6cf4f01d7"}
Nov 03 13:16:10.456611 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.456559    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 13:16:10.587931 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.587893    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\" (UniqueName: \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.588125 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.587944    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\" (UniqueName: \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.588125 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.588000    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.588432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.588132    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.588432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.588169    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.588432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.588259    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\") pod \"7919092e-f3e8-491a-987c-303bca4a3486\" (UID: \"7919092e-f3e8-491a-987c-303bca4a3486\") "
Nov 03 13:16:10.590995 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.590947    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.590995 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.590979    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "kube-api-access-czrxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:16:10.591713 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.591682    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.591811 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.591751    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.591913 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.591898    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:16:10.593688 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.593655    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs" pod "7919092e-f3e8-491a-987c-303bca4a3486" (UID: "7919092e-f3e8-491a-987c-303bca4a3486"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:16:10.691819 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691787    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.691819 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691814    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.691819 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691825    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.692042 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691833    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7919092e-f3e8-491a-987c-303bca4a3486-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.692042 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691843    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czrxd\" (UniqueName: \"kubernetes.io/projected/7919092e-f3e8-491a-987c-303bca4a3486-kube-api-access-czrxd\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:10.692042 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:10.691851    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7919092e-f3e8-491a-987c-303bca4a3486-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:16:11.409849 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.409818    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"
Nov 03 13:16:11.410292 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.409850    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p" event={"ID":"7919092e-f3e8-491a-987c-303bca4a3486","Type":"ContainerDied","Data":"4a88093f1385ccd0225ff2be493efc95dc005a0105e4ad7cfea7f79860fb2250"}
Nov 03 13:16:11.410292 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.409905    5784 scope.go:122] "RemoveContainer" containerID="fe17dbcb1fb65c8bed158fa32f3a46b628fa86d418054f93c06bc3a6cf4f01d7"
Nov 03 13:16:11.549562 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.549522    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:11.562388 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.562352    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 13:16:11.566403 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.566365    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bb1b51/distinct-w3-bb1b51-kserve-7c6b9c9bc-7pr4p"]
Nov 03 13:16:11.682018 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:11.681945    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7919092e-f3e8-491a-987c-303bca4a3486" path="/var/lib/kubelet/pods/7919092e-f3e8-491a-987c-303bca4a3486/volumes"
Nov 03 13:16:12.678554 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:12.678511    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:16.016606 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:16.016572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:19.023706 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:19.023667    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:19.192554 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:19.192516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:19.382664 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:19.382633    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:19.551980 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:19.551934    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:20.104573 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:20.104541    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:20.825876 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:20.825842    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:21.190464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:21.190419    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:21.657518 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:21.657486    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:22.439692 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:22.439648    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:25.816552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:25.816513    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:26.470844 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:16:26.470808    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:16:26.612180 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:26.612147    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:28.795267 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:28.795230    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:29.681927 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:29.681893    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:29.838361 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:29.838311    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:30.500211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:30.500172    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:34.473369 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:34.473313    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:34.598777 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:34.598740    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:16:34.647763 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:34.647723    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:16:35.114342 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:35.114310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:38.253433 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:38.253404    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:39.609261 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:39.609218    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:41.444511 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:41.444481    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:44.126492 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:44.126460    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:45.645424 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:45.645391    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:45.903744 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:45.903657    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 13:16:46.001251 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:46.001222    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 13:16:46.276786 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:46.276697    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:46.432947 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:46.432912    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:47.330556 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:47.330512    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:47.515564 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:47.515532    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:47.704722 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:47.704684    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:47.874722 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:47.874689    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:48.224473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:48.224425    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:50.093361 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:50.093328    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:50.762702 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:50.762660    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:50.904217 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:50.904117    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing" containerID="cri-o://c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599" gracePeriod=25
Nov 03 13:16:51.001687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:51.001645    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing" containerID="cri-o://7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432" gracePeriod=25
Nov 03 13:16:52.273967 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:52.273929    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:52.649464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:52.649415    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:52.812490 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:52.812444    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:54.073807 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:54.073773    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:54.524002 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:54.523967    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:54.903484 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:54.903444    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:56.471521 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:16:56.471484    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:16:56.575210 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:56.575179    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:57.291049 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:57.291016    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 13:16:57.395781 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:57.395746    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 13:16:57.685612 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:57.685573    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:57.852816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:57.852773    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:59.023031 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:59.022992    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:59.023031 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:59.023007    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:16:59.535134 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:59.535101    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:16:59.878246 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:16:59.878211    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:02.291654 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.291604    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" podUID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" containerName="payload-processing" containerID="cri-o://5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203" gracePeriod=25
Nov 03 13:17:02.396639 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.396586    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" podUID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" containerName="payload-pre-processing" containerID="cri-o://e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f" gracePeriod=25
Nov 03 13:17:02.446036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.445994    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:02.572323 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.572296    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 13:17:02.648827 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.648794    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\" (UniqueName: \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\") pod \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") "
Nov 03 13:17:02.648962 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.648901    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\") pod \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\" (UID: \"1518f5d4-a5dc-4141-a297-e52ad0cf5d80\") "
Nov 03 13:17:02.649283 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.649254    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume" pod "1518f5d4-a5dc-4141-a297-e52ad0cf5d80" (UID: "1518f5d4-a5dc-4141-a297-e52ad0cf5d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:02.650956 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.650932    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq" pod "1518f5d4-a5dc-4141-a297-e52ad0cf5d80" (UID: "1518f5d4-a5dc-4141-a297-e52ad0cf5d80"). InnerVolumeSpecName "kube-api-access-b88sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:02.658446 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.658424    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 13:17:02.750218 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.750188    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\" (UniqueName: \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\") pod \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") "
Nov 03 13:17:02.750432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.750326    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\") pod \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\" (UID: \"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565\") "
Nov 03 13:17:02.750774 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.750745    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume" pod "fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" (UID: "fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:02.751159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.750981    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88sq\" (UniqueName: \"kubernetes.io/projected/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-kube-api-access-b88sq\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:02.751159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.751005    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1518f5d4-a5dc-4141-a297-e52ad0cf5d80-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:02.751159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.751019    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:02.752885 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.752854    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47" pod "fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" (UID: "fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565"). InnerVolumeSpecName "kube-api-access-vmw47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:02.762594 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.762562    5784 generic.go:356] "Generic (PLEG): container finished" podID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" containerID="e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f" exitCode=0
Nov 03 13:17:02.762703 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.762631    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" event={"ID":"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565","Type":"ContainerDied","Data":"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"}
Nov 03 13:17:02.762703 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.762674    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"
Nov 03 13:17:02.762703 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.762691    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb" event={"ID":"fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565","Type":"ContainerDied","Data":"bcec64174798ec302491e09a6428aa628459024a9a778f9afb89080014a25bbf"}
Nov 03 13:17:02.762882 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.762721    5784 scope.go:122] "RemoveContainer" containerID="e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"
Nov 03 13:17:02.764311 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.764286    5784 generic.go:356] "Generic (PLEG): container finished" podID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" containerID="5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203" exitCode=0
Nov 03 13:17:02.764441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.764354    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"
Nov 03 13:17:02.764441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.764356    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" event={"ID":"1518f5d4-a5dc-4141-a297-e52ad0cf5d80","Type":"ContainerDied","Data":"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"}
Nov 03 13:17:02.764441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.764420    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv" event={"ID":"1518f5d4-a5dc-4141-a297-e52ad0cf5d80","Type":"ContainerDied","Data":"fa9dc357f5184dfe94e9586c8ac6d0694a74873d5424ae21e5cb732f7801de9f"}
Nov 03 13:17:02.773566 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.773549    5784 scope.go:122] "RemoveContainer" containerID="e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"
Nov 03 13:17:02.773901 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:02.773865    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f\": container with ID starting with e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f not found: ID does not exist" containerID="e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"
Nov 03 13:17:02.773983 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.773908    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f"} err="failed to get container status \"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f\": rpc error: code = NotFound desc = could not find container \"e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f\": container with ID starting with e2cd213d67750104525afd4360736b2ab558994e7ffec9f784f71548e0f4ec3f not found: ID does not exist"
Nov 03 13:17:02.773983 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.773923    5784 scope.go:122] "RemoveContainer" containerID="5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"
Nov 03 13:17:02.782912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.782896    5784 scope.go:122] "RemoveContainer" containerID="5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"
Nov 03 13:17:02.783199 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:02.783182    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203\": container with ID starting with 5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203 not found: ID does not exist" containerID="5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"
Nov 03 13:17:02.783252 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.783205    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203"} err="failed to get container status \"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203\": rpc error: code = NotFound desc = could not find container \"5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203\": container with ID starting with 5fdc86ce70850474ae543aa5ff3dfe8fdb927a3c834c31da82a9b8a517a43203 not found: ID does not exist"
Nov 03 13:17:02.852674 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.852590    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmw47\" (UniqueName: \"kubernetes.io/projected/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565-kube-api-access-vmw47\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:02.891332 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.891300    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 13:17:02.895393 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.895358    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-02f899-7d8fb8f8cf-9tvlb"]
Nov 03 13:17:02.999341 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:02.999307    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 13:17:03.003751 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:03.003722    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-02f899-5cc7f8766c-jpzhv"]
Nov 03 13:17:03.681711 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:03.681677    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" path="/var/lib/kubelet/pods/1518f5d4-a5dc-4141-a297-e52ad0cf5d80/volumes"
Nov 03 13:17:03.682102 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:03.682050    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" path="/var/lib/kubelet/pods/fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565/volumes"
Nov 03 13:17:03.890945 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:03.890913    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:05.512547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:05.512508    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:07.681931 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:07.681900    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:07.846730 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:07.846698    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:08.379869 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:08.379836    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:09.021986 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:09.021950    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:09.022370 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:09.021991    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 13:17:09.141652 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:09.141613    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:09.635204 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:09.635141    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:09.864349 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:09.864316    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:10.094581 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:10.094498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:10.368909 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:10.368826    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:11.179172 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.178572    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 13:17:11.238816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.238789    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\" (UniqueName: \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\") pod \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") "
Nov 03 13:17:11.238962 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.238864    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\") pod \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\" (UID: \"89e0b7e1-c4c1-45db-abd6-669fc5e83a44\") "
Nov 03 13:17:11.239293 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.239268    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume" pod "89e0b7e1-c4c1-45db-abd6-669fc5e83a44" (UID: "89e0b7e1-c4c1-45db-abd6-669fc5e83a44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:11.240954 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.240933    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97" pod "89e0b7e1-c4c1-45db-abd6-669fc5e83a44" (UID: "89e0b7e1-c4c1-45db-abd6-669fc5e83a44"). InnerVolumeSpecName "kube-api-access-rms97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:11.245013 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.244994    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 13:17:11.339657 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.339625    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\" (UniqueName: \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\") pod \"6ddced03-6163-4a21-9c27-7b5aa2784db1\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") "
Nov 03 13:17:11.339835 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.339724    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\") pod \"6ddced03-6163-4a21-9c27-7b5aa2784db1\" (UID: \"6ddced03-6163-4a21-9c27-7b5aa2784db1\") "
Nov 03 13:17:11.340095 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.340062    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume" pod "6ddced03-6163-4a21-9c27-7b5aa2784db1" (UID: "6ddced03-6163-4a21-9c27-7b5aa2784db1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:11.340180 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.340144    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:11.340180 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.340158    5784 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ddced03-6163-4a21-9c27-7b5aa2784db1-plugins-config-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:11.340180 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.340167    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rms97\" (UniqueName: \"kubernetes.io/projected/89e0b7e1-c4c1-45db-abd6-669fc5e83a44-kube-api-access-rms97\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:11.341942 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.341924    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r" pod "6ddced03-6163-4a21-9c27-7b5aa2784db1" (UID: "6ddced03-6163-4a21-9c27-7b5aa2784db1"). InnerVolumeSpecName "kube-api-access-nb69r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:11.440784 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.440698    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb69r\" (UniqueName: \"kubernetes.io/projected/6ddced03-6163-4a21-9c27-7b5aa2784db1-kube-api-access-nb69r\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:11.829363 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.829328    5784 generic.go:356] "Generic (PLEG): container finished" podID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerID="7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432" exitCode=0
Nov 03 13:17:11.829560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.829397    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"
Nov 03 13:17:11.829560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.829424    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" event={"ID":"6ddced03-6163-4a21-9c27-7b5aa2784db1","Type":"ContainerDied","Data":"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"}
Nov 03 13:17:11.829560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.829477    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt" event={"ID":"6ddced03-6163-4a21-9c27-7b5aa2784db1","Type":"ContainerDied","Data":"d391b66d643fd02f9c9b2ba5c6761d4da25201a38432a81b2c0542c803e4a942"}
Nov 03 13:17:11.829560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.829497    5784 scope.go:122] "RemoveContainer" containerID="7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"
Nov 03 13:17:11.830862 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.830832    5784 generic.go:356] "Generic (PLEG): container finished" podID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerID="c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599" exitCode=0
Nov 03 13:17:11.831005 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.830872    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"
Nov 03 13:17:11.831005 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.830918    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" event={"ID":"89e0b7e1-c4c1-45db-abd6-669fc5e83a44","Type":"ContainerDied","Data":"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"}
Nov 03 13:17:11.831005 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.830979    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh" event={"ID":"89e0b7e1-c4c1-45db-abd6-669fc5e83a44","Type":"ContainerDied","Data":"05d0159729893939453243dc0c82d0e1e5d1a88d8605a1d09822ea4f52476142"}
Nov 03 13:17:11.839173 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.839151    5784 scope.go:122] "RemoveContainer" containerID="7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"
Nov 03 13:17:11.839540 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:11.839522    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432\": container with ID starting with 7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432 not found: ID does not exist" containerID="7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"
Nov 03 13:17:11.839610 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.839548    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432"} err="failed to get container status \"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432\": rpc error: code = NotFound desc = could not find container \"7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432\": container with ID starting with 7cb3d3fc31e040e874ca5f47333f0bdfb48776eefca09961c002b3f84f2e0432 not found: ID does not exist"
Nov 03 13:17:11.839610 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.839563    5784 scope.go:122] "RemoveContainer" containerID="c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"
Nov 03 13:17:11.848402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.848360    5784 scope.go:122] "RemoveContainer" containerID="c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"
Nov 03 13:17:11.848666 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:11.848645    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599\": container with ID starting with c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599 not found: ID does not exist" containerID="c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"
Nov 03 13:17:11.848746 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.848673    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599"} err="failed to get container status \"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599\": rpc error: code = NotFound desc = could not find container \"c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599\": container with ID starting with c617b19ea5cbec6a862a82f42fb884117a7038a3f11cf6cc9e7ebe15b72d4599 not found: ID does not exist"
Nov 03 13:17:11.984262 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:11.984189    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 13:17:12.030454 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:12.030424    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a4489c-b6ff94777-vf7jh"]
Nov 03 13:17:12.096919 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:12.096885    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 13:17:12.099536 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:12.099510    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a4489c-5fdd8fd4bc-j6tmt"]
Nov 03 13:17:13.682547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:13.682515    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" path="/var/lib/kubelet/pods/6ddced03-6163-4a21-9c27-7b5aa2784db1/volumes"
Nov 03 13:17:13.682932 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:13.682893    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" path="/var/lib/kubelet/pods/89e0b7e1-c4c1-45db-abd6-669fc5e83a44/volumes"
Nov 03 13:17:14.455842 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.455802    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 13:17:14.456089 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.456063    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" podUID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" containerName="maas-api" containerID="cri-o://39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270" gracePeriod=30
Nov 03 13:17:14.578255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.578224    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:14.727534 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.727502    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 13:17:14.775245 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.775215    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\") pod \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") "
Nov 03 13:17:14.775479 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.775295    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") pod \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") "
Nov 03 13:17:14.775479 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.775320    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\") pod \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") "
Nov 03 13:17:14.775479 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.775410    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\" (UniqueName: \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\") pod \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\" (UID: \"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795\") "
Nov 03 13:17:14.775838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.775809    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca" pod "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" (UID: "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:14.777529 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.777502    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs" pod "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" (UID: "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:14.777662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.777615    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv" pod "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" (UID: "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795"). InnerVolumeSpecName "kube-api-access-25mfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:14.777662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.777640    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls" pod "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" (UID: "9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:14.859935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.856580    5784 generic.go:356] "Generic (PLEG): container finished" podID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" containerID="39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270" exitCode=0
Nov 03 13:17:14.859935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.856752    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" event={"ID":"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795","Type":"ContainerDied","Data":"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"}
Nov 03 13:17:14.859935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.856805    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t" event={"ID":"9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795","Type":"ContainerDied","Data":"d34e19b4f2ab8a8d3ac063725900b9c5ddd1e4fe674020dc62cbf98bad805287"}
Nov 03 13:17:14.859935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.856833    5784 scope.go:122] "RemoveContainer" containerID="39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"
Nov 03 13:17:14.859935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.856953    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"
Nov 03 13:17:14.877447 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877419    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mfv\" (UniqueName: \"kubernetes.io/projected/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-kube-api-access-25mfv\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:14.877547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877453    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:14.877547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877469    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:14.877547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877483    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:14.877547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877528    5784 scope.go:122] "RemoveContainer" containerID="39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"
Nov 03 13:17:14.877905 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:14.877878    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270\": container with ID starting with 39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270 not found: ID does not exist" containerID="39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"
Nov 03 13:17:14.877968 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:14.877918    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270"} err="failed to get container status \"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270\": rpc error: code = NotFound desc = could not find container \"39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270\": container with ID starting with 39b70c1fdee6dabc07b4c8cfd4f7d929610c01bdf9903dc55a475ece963a2270 not found: ID does not exist"
Nov 03 13:17:15.029891 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:15.029803    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 13:17:15.038479 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:15.037995    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-02f899-66bdc69656-z497t"]
Nov 03 13:17:15.229887 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:15.229843    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:15.681268 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:15.681235    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" path="/var/lib/kubelet/pods/9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795/volumes"
Nov 03 13:17:16.262766 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:16.262730    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:17.186856 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:17.186814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:17.374498 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:17.374462    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:18.653024 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:18.652985    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:19.175405 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:19.175364    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:19.861162 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:19.861115    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:21.023799 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:21.023755    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:21.329525 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:21.329440    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:24.997425 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:24.997388    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 13:17:24.997844 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:24.997580    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" podUID="382baf3a-d92f-4238-813c-01fcd9b0cc69" containerName="maas-api" containerID="cri-o://c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d" gracePeriod=30
Nov 03 13:17:25.118080 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.118042    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:25.290473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.290438    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 13:17:25.386853 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.386824    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\") pod \"382baf3a-d92f-4238-813c-01fcd9b0cc69\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") "
Nov 03 13:17:25.387052 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.386881    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\") pod \"382baf3a-d92f-4238-813c-01fcd9b0cc69\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") "
Nov 03 13:17:25.387052 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.386903    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\" (UniqueName: \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\") pod \"382baf3a-d92f-4238-813c-01fcd9b0cc69\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") "
Nov 03 13:17:25.387052 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.386950    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") pod \"382baf3a-d92f-4238-813c-01fcd9b0cc69\" (UID: \"382baf3a-d92f-4238-813c-01fcd9b0cc69\") "
Nov 03 13:17:25.387300 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.387267    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca" pod "382baf3a-d92f-4238-813c-01fcd9b0cc69" (UID: "382baf3a-d92f-4238-813c-01fcd9b0cc69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:25.389029 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.389006    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs" pod "382baf3a-d92f-4238-813c-01fcd9b0cc69" (UID: "382baf3a-d92f-4238-813c-01fcd9b0cc69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:25.389117 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.389095    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5" pod "382baf3a-d92f-4238-813c-01fcd9b0cc69" (UID: "382baf3a-d92f-4238-813c-01fcd9b0cc69"). InnerVolumeSpecName "kube-api-access-88qj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:25.389160 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.389092    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls" pod "382baf3a-d92f-4238-813c-01fcd9b0cc69" (UID: "382baf3a-d92f-4238-813c-01fcd9b0cc69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:25.488918 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.488886    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:25.488918 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.488916    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/382baf3a-d92f-4238-813c-01fcd9b0cc69-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:25.488918 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.488928    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qj5\" (UniqueName: \"kubernetes.io/projected/382baf3a-d92f-4238-813c-01fcd9b0cc69-kube-api-access-88qj5\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:25.489165 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.488942    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/382baf3a-d92f-4238-813c-01fcd9b0cc69-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:25.928879 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.928844    5784 generic.go:356] "Generic (PLEG): container finished" podID="382baf3a-d92f-4238-813c-01fcd9b0cc69" containerID="c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d" exitCode=0
Nov 03 13:17:25.929084 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.928900    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"
Nov 03 13:17:25.929084 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.928933    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" event={"ID":"382baf3a-d92f-4238-813c-01fcd9b0cc69","Type":"ContainerDied","Data":"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"}
Nov 03 13:17:25.929084 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.928997    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj" event={"ID":"382baf3a-d92f-4238-813c-01fcd9b0cc69","Type":"ContainerDied","Data":"0c155446b1636398ef1a7da98e74657d3b2eacf79f7548ed0cdb17f64135c536"}
Nov 03 13:17:25.929084 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.929027    5784 scope.go:122] "RemoveContainer" containerID="c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"
Nov 03 13:17:25.939150 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.939135    5784 scope.go:122] "RemoveContainer" containerID="c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"
Nov 03 13:17:25.939414 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:25.939396    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d\": container with ID starting with c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d not found: ID does not exist" containerID="c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"
Nov 03 13:17:25.939488 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:25.939427    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d"} err="failed to get container status \"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d\": rpc error: code = NotFound desc = could not find container \"c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d\": container with ID starting with c7758b2bde68d3b5dff6db4e48e6ca9b830a68f27935555370516d44a48c8f5d not found: ID does not exist"
Nov 03 13:17:26.081592 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:26.081547    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 13:17:26.085576 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:26.085551    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a4489c-76fbb5d96b-gqntj"]
Nov 03 13:17:26.472461 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:26.472424    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:17:26.898903 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:26.898861    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:27.090074 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:27.090038    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:27.682498 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:27.682462    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382baf3a-d92f-4238-813c-01fcd9b0cc69" path="/var/lib/kubelet/pods/382baf3a-d92f-4238-813c-01fcd9b0cc69/volumes"
Nov 03 13:17:27.827714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:27.827683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:28.803359 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:28.803325    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:28.956516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:28.956483    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:31.204645 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:31.204596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:31.363591 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:31.363556    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:31.861315 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:31.861284    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:32.602901 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:32.602866    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:33.649281 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:33.649217    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:34.713985 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:34.713952    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:35.247287 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:35.247249    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:35.960643 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:35.960602    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:36.397351 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:36.397315    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:36.558049 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:36.557976    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:39.577611 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:39.577572    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:40.619733 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:40.619700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:41.867578 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:41.867540    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:42.316711 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:42.316667    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:42.871833 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:42.871799    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:43.042705 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:43.042675    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:45.223698 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:45.223666    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:45.380574 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:45.380536    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:46.925540 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:46.925502    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:47.726053 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:47.726015    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:51.418136 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:51.418100    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:51.592647 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:51.592596    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:52.761701 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:52.761658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:55.309281 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.309230    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:55.338101 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.338062    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 13:17:55.338351 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.338326    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" podUID="6413238b-2d42-4ae0-ad5d-5e281f979903" containerName="maas-api" containerID="cri-o://7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20" gracePeriod=30
Nov 03 13:17:55.512330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.512296    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:55.593364 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.593331    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 13:17:55.713606 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.713575    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") pod \"6413238b-2d42-4ae0-ad5d-5e281f979903\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") "
Nov 03 13:17:55.713776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.713635    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\") pod \"6413238b-2d42-4ae0-ad5d-5e281f979903\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") "
Nov 03 13:17:55.713776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.713672    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\") pod \"6413238b-2d42-4ae0-ad5d-5e281f979903\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") "
Nov 03 13:17:55.713776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.713759    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\" (UniqueName: \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\") pod \"6413238b-2d42-4ae0-ad5d-5e281f979903\" (UID: \"6413238b-2d42-4ae0-ad5d-5e281f979903\") "
Nov 03 13:17:55.714308 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.714278    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca" pod "6413238b-2d42-4ae0-ad5d-5e281f979903" (UID: "6413238b-2d42-4ae0-ad5d-5e281f979903"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:17:55.716805 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.716762    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls" pod "6413238b-2d42-4ae0-ad5d-5e281f979903" (UID: "6413238b-2d42-4ae0-ad5d-5e281f979903"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:55.739584 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.739523    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs" pod "6413238b-2d42-4ae0-ad5d-5e281f979903" (UID: "6413238b-2d42-4ae0-ad5d-5e281f979903"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:17:55.740173 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.740143    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg" pod "6413238b-2d42-4ae0-ad5d-5e281f979903" (UID: "6413238b-2d42-4ae0-ad5d-5e281f979903"). InnerVolumeSpecName "kube-api-access-czlkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:17:55.815687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.815591    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:55.815687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.815629    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6413238b-2d42-4ae0-ad5d-5e281f979903-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:55.815687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.815647    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6413238b-2d42-4ae0-ad5d-5e281f979903-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:55.815687 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.815663    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czlkg\" (UniqueName: \"kubernetes.io/projected/6413238b-2d42-4ae0-ad5d-5e281f979903-kube-api-access-czlkg\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:17:55.816219 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:55.816191    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:56.118533 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.118448    5784 generic.go:356] "Generic (PLEG): container finished" podID="6413238b-2d42-4ae0-ad5d-5e281f979903" containerID="7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20" exitCode=0
Nov 03 13:17:56.118533 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.118495    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" event={"ID":"6413238b-2d42-4ae0-ad5d-5e281f979903","Type":"ContainerDied","Data":"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"}
Nov 03 13:17:56.118743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.118558    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq" event={"ID":"6413238b-2d42-4ae0-ad5d-5e281f979903","Type":"ContainerDied","Data":"77e0c6a648fa7a25d4ca84f9c8708ec7dd8d953836b5846ee486055944c92d69"}
Nov 03 13:17:56.118743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.118510    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"
Nov 03 13:17:56.118743 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.118588    5784 scope.go:122] "RemoveContainer" containerID="7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"
Nov 03 13:17:56.129870 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.129852    5784 scope.go:122] "RemoveContainer" containerID="7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"
Nov 03 13:17:56.130132 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:56.130112    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20\": container with ID starting with 7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20 not found: ID does not exist" containerID="7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"
Nov 03 13:17:56.130180 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.130141    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20"} err="failed to get container status \"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20\": rpc error: code = NotFound desc = could not find container \"7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20\": container with ID starting with 7e2a7017be4b359b1b7872176112a295c40a17675883a58d603c1caf4ad7dc20 not found: ID does not exist"
Nov 03 13:17:56.273346 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.273312    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 13:17:56.283197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:56.283168    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-94e247-9c7cf894-rg5vq"]
Nov 03 13:17:56.472832 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:17:56.472794    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:17:57.214009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:57.213979    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:57.680908 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:57.680870    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6413238b-2d42-4ae0-ad5d-5e281f979903" path="/var/lib/kubelet/pods/6413238b-2d42-4ae0-ad5d-5e281f979903/volumes"
Nov 03 13:17:58.757570 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:58.757511    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:17:58.916278 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:17:58.916240    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:01.333244 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.333200    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 13:18:01.333798 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.333465    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" podUID="2d63a51c-bf03-4100-addc-0264a9058016" containerName="maas-api" containerID="cri-o://65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654" gracePeriod=30
Nov 03 13:18:01.477095 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.477056    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:01.583753 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.583694    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 13:18:01.673993 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.673960    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\") pod \"2d63a51c-bf03-4100-addc-0264a9058016\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") "
Nov 03 13:18:01.674184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.674078    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\") pod \"2d63a51c-bf03-4100-addc-0264a9058016\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") "
Nov 03 13:18:01.674184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.674178    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\" (UniqueName: \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\") pod \"2d63a51c-bf03-4100-addc-0264a9058016\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") "
Nov 03 13:18:01.674323 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.674241    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") pod \"2d63a51c-bf03-4100-addc-0264a9058016\" (UID: \"2d63a51c-bf03-4100-addc-0264a9058016\") "
Nov 03 13:18:01.674481 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.674458    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca" pod "2d63a51c-bf03-4100-addc-0264a9058016" (UID: "2d63a51c-bf03-4100-addc-0264a9058016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:01.674634 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.674615    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d63a51c-bf03-4100-addc-0264a9058016-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.676127 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.676097    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs" pod "2d63a51c-bf03-4100-addc-0264a9058016" (UID: "2d63a51c-bf03-4100-addc-0264a9058016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:01.676244 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.676217    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls" pod "2d63a51c-bf03-4100-addc-0264a9058016" (UID: "2d63a51c-bf03-4100-addc-0264a9058016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:01.676596 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.676574    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9" pod "2d63a51c-bf03-4100-addc-0264a9058016" (UID: "2d63a51c-bf03-4100-addc-0264a9058016"). InnerVolumeSpecName "kube-api-access-jwkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:01.767911 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.767880    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:01.776143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.776119    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwkb9\" (UniqueName: \"kubernetes.io/projected/2d63a51c-bf03-4100-addc-0264a9058016-kube-api-access-jwkb9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.776143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.776143    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:01.776272 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:01.776154    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d63a51c-bf03-4100-addc-0264a9058016-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:02.159809 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.159767    5784 generic.go:356] "Generic (PLEG): container finished" podID="2d63a51c-bf03-4100-addc-0264a9058016" containerID="65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654" exitCode=0
Nov 03 13:18:02.159979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.159812    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"
Nov 03 13:18:02.159979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.159836    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" event={"ID":"2d63a51c-bf03-4100-addc-0264a9058016","Type":"ContainerDied","Data":"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"}
Nov 03 13:18:02.159979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.159894    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595" event={"ID":"2d63a51c-bf03-4100-addc-0264a9058016","Type":"ContainerDied","Data":"c52ce2f73240a11e0f767a00b1166e5ee337c2f934567b4c80f7e0573bd40abc"}
Nov 03 13:18:02.159979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.159924    5784 scope.go:122] "RemoveContainer" containerID="65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"
Nov 03 13:18:02.169419 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.169394    5784 scope.go:122] "RemoveContainer" containerID="65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"
Nov 03 13:18:02.169701 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:02.169681    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654\": container with ID starting with 65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654 not found: ID does not exist" containerID="65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"
Nov 03 13:18:02.169776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.169715    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654"} err="failed to get container status \"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654\": rpc error: code = NotFound desc = could not find container \"65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654\": container with ID starting with 65b1fc406fc96d916f9e8c67405f59d3f867b1dc746ab514c9ce106ed6e09654 not found: ID does not exist"
Nov 03 13:18:02.313648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.313614    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 13:18:02.359667 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:02.359637    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-6fb3a8-749988494d-q5595"]
Nov 03 13:18:03.121640 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:03.121601    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:03.682558 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:03.682523    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d63a51c-bf03-4100-addc-0264a9058016" path="/var/lib/kubelet/pods/2d63a51c-bf03-4100-addc-0264a9058016/volumes"
Nov 03 13:18:05.338043 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:05.338007    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:05.498437 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:05.498397    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:06.024699 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:06.024636    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:06.186237 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:06.186204    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:06.827326 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:06.827295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:07.326855 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:07.326820    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:07.854432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:07.854403    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:08.063900 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:08.063870    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:08.688858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:08.688819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:10.746330 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:10.746294    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:10.936238 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:10.936198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:12.107227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:12.107197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:12.572648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:12.572615    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:12.790163 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:12.790132    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:13.090735 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:13.090704    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:14.486689 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:14.486658    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:14.885582 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:14.885551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:15.042858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:15.042825    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:17.755411 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:17.755361    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:17.956142 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:17.956106    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 13:18:18.063254 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:18.063150    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:19.969773 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:19.969740    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.533121 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.533081    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 13:18:21.533697 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.533329    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" containerName="maas-api" containerID="cri-o://d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c" gracePeriod=30
Nov 03 13:18:21.570772 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.570731    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.112:8443/health\": dial tcp 10.134.0.112:8443: connect: connection refused"
Nov 03 13:18:21.662779 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.662741    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:21.809696 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.809667    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 13:18:21.883402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.883344    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\") pod \"0b91d248-e578-4e1c-a954-d281e4c66775\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") "
Nov 03 13:18:21.883402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.883407    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\" (UniqueName: \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\") pod \"0b91d248-e578-4e1c-a954-d281e4c66775\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") "
Nov 03 13:18:21.883659 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.883468    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\") pod \"0b91d248-e578-4e1c-a954-d281e4c66775\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") "
Nov 03 13:18:21.883659 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.883492    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\") pod \"0b91d248-e578-4e1c-a954-d281e4c66775\" (UID: \"0b91d248-e578-4e1c-a954-d281e4c66775\") "
Nov 03 13:18:21.883840 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.883803    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca" pod "0b91d248-e578-4e1c-a954-d281e4c66775" (UID: "0b91d248-e578-4e1c-a954-d281e4c66775"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:21.885467 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.885442    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls" pod "0b91d248-e578-4e1c-a954-d281e4c66775" (UID: "0b91d248-e578-4e1c-a954-d281e4c66775"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:21.885561 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.885469    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs" pod "0b91d248-e578-4e1c-a954-d281e4c66775" (UID: "0b91d248-e578-4e1c-a954-d281e4c66775"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:21.885601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.885566    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4" pod "0b91d248-e578-4e1c-a954-d281e4c66775" (UID: "0b91d248-e578-4e1c-a954-d281e4c66775"). InnerVolumeSpecName "kube-api-access-6sdm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:21.985168 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.985134    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:21.985168 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.985161    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b91d248-e578-4e1c-a954-d281e4c66775-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:21.985168 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.985172    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdm4\" (UniqueName: \"kubernetes.io/projected/0b91d248-e578-4e1c-a954-d281e4c66775-kube-api-access-6sdm4\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:21.985472 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:21.985182    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b91d248-e578-4e1c-a954-d281e4c66775-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:22.291018 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.290983    5784 generic.go:356] "Generic (PLEG): container finished" podID="0b91d248-e578-4e1c-a954-d281e4c66775" containerID="d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c" exitCode=0
Nov 03 13:18:22.291184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.291028    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"
Nov 03 13:18:22.291184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.291068    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" event={"ID":"0b91d248-e578-4e1c-a954-d281e4c66775","Type":"ContainerDied","Data":"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"}
Nov 03 13:18:22.291184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.291120    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9" event={"ID":"0b91d248-e578-4e1c-a954-d281e4c66775","Type":"ContainerDied","Data":"cbab14a440d5c2c70891d769aed0b3e26261196a4e1b35d1b97ad8e2a58c13a7"}
Nov 03 13:18:22.291184 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.291143    5784 scope.go:122] "RemoveContainer" containerID="d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"
Nov 03 13:18:22.302871 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.302847    5784 scope.go:122] "RemoveContainer" containerID="d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"
Nov 03 13:18:22.303167 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:22.303145    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c\": container with ID starting with d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c not found: ID does not exist" containerID="d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"
Nov 03 13:18:22.303263 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.303174    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c"} err="failed to get container status \"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c\": rpc error: code = NotFound desc = could not find container \"d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c\": container with ID starting with d1882ca2f9522b6aea38efa3e63fc846da6838f4721594baea1368902c9ade8c not found: ID does not exist"
Nov 03 13:18:22.444745 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.444712    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 13:18:22.490654 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:22.490617    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-ab27b8-78995fdf5b-lsps9"]
Nov 03 13:18:23.341192 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:23.341142    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:23.594245 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:23.594152    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:23.682440 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:23.682406    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" path="/var/lib/kubelet/pods/0b91d248-e578-4e1c-a954-d281e4c66775/volumes"
Nov 03 13:18:25.701565 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:25.701513    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:25.865249 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:25.865214    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:26.473262 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:26.473222    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:18:27.109369 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:27.109340    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:27.621455 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:27.621408    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:27.821935 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:27.821900    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:28.380628 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:28.380591    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:29.672617 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:29.672564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:29.836430 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:29.836393    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:32.677123 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:32.677092    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:32.977252 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:32.977164    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main" containerID="cri-o://2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd" gracePeriod=45
Nov 03 13:18:33.235567 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.235504    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 13:18:33.367726 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.367693    5784 generic.go:356] "Generic (PLEG): container finished" podID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerID="2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd" exitCode=0
Nov 03 13:18:33.367938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.367751    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" event={"ID":"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf","Type":"ContainerDied","Data":"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"}
Nov 03 13:18:33.367938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.367762    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"
Nov 03 13:18:33.367938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.367781    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp" event={"ID":"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf","Type":"ContainerDied","Data":"f217b57c61a875a41d118074e6134bcaa4aafbf2c80f909e93113b1e7633ab7a"}
Nov 03 13:18:33.367938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.367798    5784 scope.go:122] "RemoveContainer" containerID="2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"
Nov 03 13:18:33.378097 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.378075    5784 scope.go:122] "RemoveContainer" containerID="2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"
Nov 03 13:18:33.378412 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:33.378387    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd\": container with ID starting with 2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd not found: ID does not exist" containerID="2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"
Nov 03 13:18:33.378481 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.378423    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd"} err="failed to get container status \"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd\": rpc error: code = NotFound desc = could not find container \"2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd\": container with ID starting with 2a50facc5ef1141d01186f37e47c67e828d5cd8cc33b3e4b14d1a06dc6887bdd not found: ID does not exist"
Nov 03 13:18:33.403845 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.403820    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.403979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.403888    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.404041 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404026    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\" (UniqueName: \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.404097 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404077    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.404164 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404115    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.404164 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404116    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.404164 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404143    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.404338 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404153    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\") pod \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\" (UID: \"7f2e6d19-72f9-4327-8bd7-ea7bb992aebf\") "
Nov 03 13:18:33.404459 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404352    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.404568 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404546    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.404568 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404565    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.404756 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.404575    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.406309 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.406286    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:33.406423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.406331    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "kube-api-access-xb8bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:33.406423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.406350    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm" pod "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" (UID: "7f2e6d19-72f9-4327-8bd7-ea7bb992aebf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:18:33.505593 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.505514    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8bs\" (UniqueName: \"kubernetes.io/projected/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-kube-api-access-xb8bs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.505593 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.505547    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.505593 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.505560    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:33.687832 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.687804    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:33.784485 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.784408    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 13:18:33.830139 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.830109    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5951bc/simulator-w1-5951bc-kserve-7679bd4d49-xhbrp"]
Nov 03 13:18:33.852603 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:33.852576    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:34.677810 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:34.677772    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.267488 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:35.267451    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:35.680676 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:35.680645    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" path="/var/lib/kubelet/pods/7f2e6d19-72f9-4327-8bd7-ea7bb992aebf/volumes"
Nov 03 13:18:36.107848 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:36.107768    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:36.268855 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:36.268808    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:37.491424 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:37.491371    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:40.241224 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:40.241185    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.431282 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.431201    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 13:18:41.431709 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.431517    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" podUID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" containerName="maas-api" containerID="cri-o://0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899" gracePeriod=30
Nov 03 13:18:41.509758 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.509725    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.698344 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.698310    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:41.716785 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.716763    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 13:18:41.892177 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.892139    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") pod \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") "
Nov 03 13:18:41.892350 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.892239    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\" (UniqueName: \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\") pod \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") "
Nov 03 13:18:41.892350 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.892270    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\") pod \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") "
Nov 03 13:18:41.892533 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.892367    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\") pod \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\" (UID: \"d9415789-a9c5-4ae5-b47f-dc943676cc5f\") "
Nov 03 13:18:41.892722 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.892687    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca" pod "d9415789-a9c5-4ae5-b47f-dc943676cc5f" (UID: "d9415789-a9c5-4ae5-b47f-dc943676cc5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:41.894398 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.894359    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt" pod "d9415789-a9c5-4ae5-b47f-dc943676cc5f" (UID: "d9415789-a9c5-4ae5-b47f-dc943676cc5f"). InnerVolumeSpecName "kube-api-access-qmqkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:41.894477 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.894393    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs" pod "d9415789-a9c5-4ae5-b47f-dc943676cc5f" (UID: "d9415789-a9c5-4ae5-b47f-dc943676cc5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:41.894477 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.894467    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls" pod "d9415789-a9c5-4ae5-b47f-dc943676cc5f" (UID: "d9415789-a9c5-4ae5-b47f-dc943676cc5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:41.993227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.993154    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9415789-a9c5-4ae5-b47f-dc943676cc5f-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:41.993227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.993180    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:41.993227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.993189    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmqkt\" (UniqueName: \"kubernetes.io/projected/d9415789-a9c5-4ae5-b47f-dc943676cc5f-kube-api-access-qmqkt\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:41.993227 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:41.993198    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9415789-a9c5-4ae5-b47f-dc943676cc5f-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:42.423591 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.423555    5784 generic.go:356] "Generic (PLEG): container finished" podID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" containerID="0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899" exitCode=0
Nov 03 13:18:42.423815 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.423608    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"
Nov 03 13:18:42.423815 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.423650    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" event={"ID":"d9415789-a9c5-4ae5-b47f-dc943676cc5f","Type":"ContainerDied","Data":"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"}
Nov 03 13:18:42.423815 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.423709    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn" event={"ID":"d9415789-a9c5-4ae5-b47f-dc943676cc5f","Type":"ContainerDied","Data":"ea9e60e9a2e9a0da4f9b4e3efd26844a5c488dcfb82a62dd2470a0bf02ab52e0"}
Nov 03 13:18:42.423815 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.423739    5784 scope.go:122] "RemoveContainer" containerID="0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"
Nov 03 13:18:42.434675 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.434486    5784 scope.go:122] "RemoveContainer" containerID="0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"
Nov 03 13:18:42.434920 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:42.434751    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899\": container with ID starting with 0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899 not found: ID does not exist" containerID="0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"
Nov 03 13:18:42.434920 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.434778    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899"} err="failed to get container status \"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899\": rpc error: code = NotFound desc = could not find container \"0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899\": container with ID starting with 0b94d8d63756adfe6dc8acc384e0ef2ce25a6ec87da2af23f3f094ab779cc899 not found: ID does not exist"
Nov 03 13:18:42.522216 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.522169    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 13:18:42.522443 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.522396    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" podUID="708f97d2-541f-4676-a00d-db59163b08bc" containerName="maas-api" containerID="cri-o://fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362" gracePeriod=30
Nov 03 13:18:42.583595 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.583565    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 13:18:42.589073 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.589042    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-743d9b-6c7fc455cb-7nxqn"]
Nov 03 13:18:42.635651 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.635617    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:42.777438 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.777411    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 13:18:42.899932 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.899900    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\") pod \"708f97d2-541f-4676-a00d-db59163b08bc\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") "
Nov 03 13:18:42.900128 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.899956    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\") pod \"708f97d2-541f-4676-a00d-db59163b08bc\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") "
Nov 03 13:18:42.900128 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.900054    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\" (UniqueName: \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\") pod \"708f97d2-541f-4676-a00d-db59163b08bc\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") "
Nov 03 13:18:42.900128 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.900113    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") pod \"708f97d2-541f-4676-a00d-db59163b08bc\" (UID: \"708f97d2-541f-4676-a00d-db59163b08bc\") "
Nov 03 13:18:42.900431 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.900405    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca" pod "708f97d2-541f-4676-a00d-db59163b08bc" (UID: "708f97d2-541f-4676-a00d-db59163b08bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:18:42.902313 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.902283    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs" pod "708f97d2-541f-4676-a00d-db59163b08bc" (UID: "708f97d2-541f-4676-a00d-db59163b08bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:42.902438 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.902340    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9" pod "708f97d2-541f-4676-a00d-db59163b08bc" (UID: "708f97d2-541f-4676-a00d-db59163b08bc"). InnerVolumeSpecName "kube-api-access-vxph9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:18:42.902438 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:42.902354    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls" pod "708f97d2-541f-4676-a00d-db59163b08bc" (UID: "708f97d2-541f-4676-a00d-db59163b08bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:18:43.001347 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.001239    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:43.001707 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.001680    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/708f97d2-541f-4676-a00d-db59163b08bc-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:43.002486 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.001711    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/708f97d2-541f-4676-a00d-db59163b08bc-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:43.002486 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.001736    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxph9\" (UniqueName: \"kubernetes.io/projected/708f97d2-541f-4676-a00d-db59163b08bc-kube-api-access-vxph9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:18:43.432175 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.432135    5784 generic.go:356] "Generic (PLEG): container finished" podID="708f97d2-541f-4676-a00d-db59163b08bc" containerID="fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362" exitCode=0
Nov 03 13:18:43.432423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.432185    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"
Nov 03 13:18:43.432423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.432214    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" event={"ID":"708f97d2-541f-4676-a00d-db59163b08bc","Type":"ContainerDied","Data":"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"}
Nov 03 13:18:43.432423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.432262    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7" event={"ID":"708f97d2-541f-4676-a00d-db59163b08bc","Type":"ContainerDied","Data":"9516740dc395087b2708524597a9118ea96e3fec18b705f649754a16c4af88ad"}
Nov 03 13:18:43.432423 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.432284    5784 scope.go:122] "RemoveContainer" containerID="fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"
Nov 03 13:18:43.445394 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.445272    5784 scope.go:122] "RemoveContainer" containerID="fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"
Nov 03 13:18:43.445688 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:43.445598    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362\": container with ID starting with fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362 not found: ID does not exist" containerID="fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"
Nov 03 13:18:43.445688 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.445625    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362"} err="failed to get container status \"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362\": rpc error: code = NotFound desc = could not find container \"fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362\": container with ID starting with fb00cd4e714f1b5d842422b1d35690e940bb88ec73615e872ea6516bfc7ba362 not found: ID does not exist"
Nov 03 13:18:43.587583 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.587545    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 13:18:43.590268 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.590246    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5951bc-57d5ccd6f7-ndvb7"]
Nov 03 13:18:43.680443 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.680409    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708f97d2-541f-4676-a00d-db59163b08bc" path="/var/lib/kubelet/pods/708f97d2-541f-4676-a00d-db59163b08bc/volumes"
Nov 03 13:18:43.680842 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:43.680827    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" path="/var/lib/kubelet/pods/d9415789-a9c5-4ae5-b47f-dc943676cc5f/volumes"
Nov 03 13:18:44.331799 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:44.331765    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:46.270296 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:46.270254    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:46.435146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:46.435117    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:47.639083 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:47.639043    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:47.806295 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:47.806255    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:48.241092 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:48.241056    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:48.703156 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:48.703121    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:49.109454 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:49.109271    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:53.630279 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:53.630243    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 13:18:53.743280 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:53.743204    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:53.912969 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:53.912862    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:55.041077 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:55.041039    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:55.203792 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:55.203762    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:55.406286 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:55.406252    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:55.565618 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:55.565574    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:56.471707 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:18:56.471675    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:18:57.841030 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:57.840990    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:58.308992 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:58.308956    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:18:58.468170 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:18:58.468137    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:01.437460 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:01.437422    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:01.602163 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:01.602133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:02.421543 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:02.421502    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:02.582877 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:02.582844    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:03.255934 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:03.255897    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:08.011960 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:08.011923    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:08.650448 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:08.650399    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" containerID="cri-o://adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5" gracePeriod=45
Nov 03 13:19:08.685359 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:08.685328    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused"
Nov 03 13:19:08.903833 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:08.903777    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 13:19:09.025624 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.025594    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:09.059484 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059454    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\" (UniqueName: \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.059674 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059525    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.059674 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059568    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\" (UniqueName: \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.059674 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059616    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.059674 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059639    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.059869 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.059723    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\") pod \"abc14fde-2904-42ca-b96e-97107c23d314\" (UID: \"abc14fde-2904-42ca-b96e-97107c23d314\") "
Nov 03 13:19:09.060248 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.060217    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.060429 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.060232    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.061183 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.061142    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.062306 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.062269    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:09.062610 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.062588    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:19:09.063076 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.063055    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r" pod "abc14fde-2904-42ca-b96e-97107c23d314" (UID: "abc14fde-2904-42ca-b96e-97107c23d314"). InnerVolumeSpecName "kube-api-access-trq6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:09.160878 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160800    5784 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-tmp-dir\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.160878 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160832    5784 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-model-cache\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.160878 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160845    5784 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abc14fde-2904-42ca-b96e-97107c23d314-tls-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.160878 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160858    5784 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-dshm\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.160878 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160872    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trq6r\" (UniqueName: \"kubernetes.io/projected/abc14fde-2904-42ca-b96e-97107c23d314-kube-api-access-trq6r\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.161151 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.160886    5784 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abc14fde-2904-42ca-b96e-97107c23d314-home\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:09.608035 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.607939    5784 generic.go:356] "Generic (PLEG): container finished" podID="abc14fde-2904-42ca-b96e-97107c23d314" containerID="adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5" exitCode=0
Nov 03 13:19:09.608035 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.607982    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" event={"ID":"abc14fde-2904-42ca-b96e-97107c23d314","Type":"ContainerDied","Data":"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"}
Nov 03 13:19:09.608035 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.607985    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"
Nov 03 13:19:09.608035 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.608019    5784 scope.go:122] "RemoveContainer" containerID="adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"
Nov 03 13:19:09.608413 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.608009    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg" event={"ID":"abc14fde-2904-42ca-b96e-97107c23d314","Type":"ContainerDied","Data":"4ea2691b65bd6b05b10496ca7043b939b169d5e04c9b0a27edeeca266be15588"}
Nov 03 13:19:09.617864 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.617835    5784 scope.go:122] "RemoveContainer" containerID="adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"
Nov 03 13:19:09.618129 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:19:09.618106    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5\": container with ID starting with adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5 not found: ID does not exist" containerID="adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"
Nov 03 13:19:09.618175 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.618138    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5"} err="failed to get container status \"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5\": rpc error: code = NotFound desc = could not find container \"adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5\": container with ID starting with adb0cc4dce898def2f679a6c4918d9d35e7b42fd203166bcdc8ef6f8fb57b0d5 not found: ID does not exist"
Nov 03 13:19:09.761223 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.761187    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 13:19:09.806072 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:09.806046    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9893f0/distinct-w0-9893f0-kserve-5db9cccfbb-6zjxg"]
Nov 03 13:19:11.680716 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:11.680677    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abc14fde-2904-42ca-b96e-97107c23d314" path="/var/lib/kubelet/pods/abc14fde-2904-42ca-b96e-97107c23d314/volumes"
Nov 03 13:19:13.715339 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:13.715267    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:17.433451 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.433410    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 13:19:17.433924 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.433658    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" podUID="d30341b2-3f0e-4970-ac6c-3f2948df548f" containerName="maas-api" containerID="cri-o://74c8b8828d73e3ca9e4b9c772b34bcdfab68028772a87b9e074f0f7903072ddf" gracePeriod=30
Nov 03 13:19:17.560836 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.560807    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:17.664255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.664030    5784 generic.go:356] "Generic (PLEG): container finished" podID="d30341b2-3f0e-4970-ac6c-3f2948df548f" containerID="74c8b8828d73e3ca9e4b9c772b34bcdfab68028772a87b9e074f0f7903072ddf" exitCode=0
Nov 03 13:19:17.664255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.664192    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" event={"ID":"d30341b2-3f0e-4970-ac6c-3f2948df548f","Type":"ContainerDied","Data":"74c8b8828d73e3ca9e4b9c772b34bcdfab68028772a87b9e074f0f7903072ddf"}
Nov 03 13:19:17.716319 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.716292    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 13:19:17.851401 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.851269    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\") pod \"d30341b2-3f0e-4970-ac6c-3f2948df548f\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") "
Nov 03 13:19:17.851401 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.851346    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\" (UniqueName: \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\") pod \"d30341b2-3f0e-4970-ac6c-3f2948df548f\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") "
Nov 03 13:19:17.851650 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.851471    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") pod \"d30341b2-3f0e-4970-ac6c-3f2948df548f\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") "
Nov 03 13:19:17.851650 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.851562    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\") pod \"d30341b2-3f0e-4970-ac6c-3f2948df548f\" (UID: \"d30341b2-3f0e-4970-ac6c-3f2948df548f\") "
Nov 03 13:19:17.852222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.852197    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca" pod "d30341b2-3f0e-4970-ac6c-3f2948df548f" (UID: "d30341b2-3f0e-4970-ac6c-3f2948df548f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:19:17.852757 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.852520    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d30341b2-3f0e-4970-ac6c-3f2948df548f-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:17.853996 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.853959    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs" pod "d30341b2-3f0e-4970-ac6c-3f2948df548f" (UID: "d30341b2-3f0e-4970-ac6c-3f2948df548f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:17.854431 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.854406    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9" pod "d30341b2-3f0e-4970-ac6c-3f2948df548f" (UID: "d30341b2-3f0e-4970-ac6c-3f2948df548f"). InnerVolumeSpecName "kube-api-access-jjrg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:19:17.854509 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.854448    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls" pod "d30341b2-3f0e-4970-ac6c-3f2948df548f" (UID: "d30341b2-3f0e-4970-ac6c-3f2948df548f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:19:17.954875 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.954789    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:17.954875 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.954825    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d30341b2-3f0e-4970-ac6c-3f2948df548f-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:17.954875 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:17.954842    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjrg9\" (UniqueName: \"kubernetes.io/projected/d30341b2-3f0e-4970-ac6c-3f2948df548f-kube-api-access-jjrg9\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:19:18.671663 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:18.671627    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"
Nov 03 13:19:18.671663 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:18.671654    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q" event={"ID":"d30341b2-3f0e-4970-ac6c-3f2948df548f","Type":"ContainerDied","Data":"1ab1d327354ce797fe7f9f3379ac6ef059339fc8dfe65b78bd2532b54d1167e5"}
Nov 03 13:19:18.672159 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:18.671713    5784 scope.go:122] "RemoveContainer" containerID="74c8b8828d73e3ca9e4b9c772b34bcdfab68028772a87b9e074f0f7903072ddf"
Nov 03 13:19:18.825691 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:18.825655    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 13:19:18.870362 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:18.870326    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9893f0-85c6c98576-m4x7q"]
Nov 03 13:19:19.438814 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:19.438778    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.601660 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:19.601599    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:19.681502 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:19.681466    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30341b2-3f0e-4970-ac6c-3f2948df548f" path="/var/lib/kubelet/pods/d30341b2-3f0e-4970-ac6c-3f2948df548f/volumes"
Nov 03 13:19:26.472746 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:19:26.472699    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:19:29.909490 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:29.909461    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:30.079601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:30.079567    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:31.117591 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:31.117553    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:34.119857 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:34.119808    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:34.652224 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:34.652195    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:35.672254 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:35.672225    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fccf55576-2gn95_e9117e63-a8c2-4c65-a7d0-adead6725e93/authorino/0.log"
Nov 03 13:19:35.937334 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:35.937258    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m9n58_f248db66-6a89-4874-92c0-d448e2380828/manager/0.log"
Nov 03 13:19:36.180866 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:36.180840    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kpq2r_7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8/manager/0.log"
Nov 03 13:19:36.736955 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:36.736920    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:36.921162 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:36.921118    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:37.174498 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:37.174461    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-db85w_7822d282-6299-48b9-8dc4-df94f622047c/manager/0.log"
Nov 03 13:19:37.827871 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:37.827839    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pw5vh_f993bb43-355f-439b-9add-9ec55f05ed03/manager/0.log"
Nov 03 13:19:39.085816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:39.085782    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-np6xb_19d9ca6d-3426-4d37-86e9-f7051fc5ce41/manager/0.log"
Nov 03 13:19:39.322191 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:39.322161    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cc94k_ce713df0-9f1c-4581-8971-6dbc92a58b67/manager/0.log"
Nov 03 13:19:39.579419 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:39.579372    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rxmxg_4e96077b-bec2-45f0-97bc-58f8f620f925/manager/0.log"
Nov 03 13:19:40.077671 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:40.077634    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-vfmn7_b9c74301-9f8c-4263-baf0-2b9b5583ff4c/manager/1.log"
Nov 03 13:19:40.924547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:40.924513    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q6gzj_4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4/proxy/0.log"
Nov 03 13:19:41.618904 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:41.618868    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78c58c9bfd-vfmn7_b9c74301-9f8c-4263-baf0-2b9b5583ff4c/manager/0.log"
Nov 03 13:19:45.922090 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:45.922057    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:46.515923 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:46.515887    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:46.979790 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:46.979760    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:49.567984 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:49.567945    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:49.742869 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:49.742827    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:49.906877 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:49.906839    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:52.769015 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:52.768983    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:52.944195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:52.944159    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:55.409097 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:55.409056    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:55.565346 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:55.565269    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:55.722979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:19:55.722950    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:19:56.470400 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:19:56.470355    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:20:00.019045 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:00.019010    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:04.538481 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:04.538445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:08.929469 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:08.929431    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:09.399195 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:09.399161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:13.727321 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:13.727228    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:18.264711 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:18.264670    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:22.931501 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:22.931398    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:26.471219 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:20:26.471181    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:20:27.325481 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:27.325448    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:31.834586 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:31.834551    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:36.188702 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:36.188656    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:37.929609 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:37.929577    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:38.078621 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:38.078589    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:40.130657 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:40.130619    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:41.524714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:41.524669    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:41.795033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:41.794945    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:47.238233 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:47.238198    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:20:51.575339 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.575302    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 13:20:51.575833 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.575549    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager" containerID="cri-o://d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40" gracePeriod=30
Nov 03 13:20:51.827642 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.827573    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 13:20:51.945468 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.945434    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\") pod \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") "
Nov 03 13:20:51.945670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.945529    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\" (UniqueName: \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\") pod \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") "
Nov 03 13:20:51.945670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.945559    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\") pod \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\" (UID: \"b9c74301-9f8c-4263-baf0-2b9b5583ff4c\") "
Nov 03 13:20:51.950414 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.948264    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs" pod "b9c74301-9f8c-4263-baf0-2b9b5583ff4c" (UID: "b9c74301-9f8c-4263-baf0-2b9b5583ff4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:51.950414 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.948586    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v" pod "b9c74301-9f8c-4263-baf0-2b9b5583ff4c" (UID: "b9c74301-9f8c-4263-baf0-2b9b5583ff4c"). InnerVolumeSpecName "kube-api-access-b2k5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:20:51.953507 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:51.953474    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert" pod "b9c74301-9f8c-4263-baf0-2b9b5583ff4c" (UID: "b9c74301-9f8c-4263-baf0-2b9b5583ff4c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:20:52.047219 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.047186    5784 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-webhook-cert\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.047219 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.047218    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2k5v\" (UniqueName: \"kubernetes.io/projected/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-kube-api-access-b2k5v\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.047219 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.047228    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9c74301-9f8c-4263-baf0-2b9b5583ff4c-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:20:52.226185 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.226147    5784 generic.go:356] "Generic (PLEG): container finished" podID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerID="d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40" exitCode=0
Nov 03 13:20:52.226402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.226219    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7"
Nov 03 13:20:52.226402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.226223    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerDied","Data":"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"}
Nov 03 13:20:52.226402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.226344    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78c58c9bfd-vfmn7" event={"ID":"b9c74301-9f8c-4263-baf0-2b9b5583ff4c","Type":"ContainerDied","Data":"4f9f53580502152a7d48dfbff99c4d964d63e552f16850fd40385431d1139375"}
Nov 03 13:20:52.226402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.226366    5784 scope.go:122] "RemoveContainer" containerID="d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"
Nov 03 13:20:52.243000 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.242977    5784 scope.go:122] "RemoveContainer" containerID="e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"
Nov 03 13:20:52.254546 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.254519    5784 scope.go:122] "RemoveContainer" containerID="d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"
Nov 03 13:20:52.254960 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:20:52.254937    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40\": container with ID starting with d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40 not found: ID does not exist" containerID="d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"
Nov 03 13:20:52.255069 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.254975    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40"} err="failed to get container status \"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40\": rpc error: code = NotFound desc = could not find container \"d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40\": container with ID starting with d9e1575ef068d728aa70652944aff11b95f67a04ea95a326d884edf4a5babb40 not found: ID does not exist"
Nov 03 13:20:52.255069 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.255001    5784 scope.go:122] "RemoveContainer" containerID="e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"
Nov 03 13:20:52.255456 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:20:52.255367    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31\": container with ID starting with e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31 not found: ID does not exist" containerID="e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"
Nov 03 13:20:52.255456 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.255429    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31"} err="failed to get container status \"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31\": rpc error: code = NotFound desc = could not find container \"e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31\": container with ID starting with e8b2aec6c5169f45ef3a86928330f5713bb92d8efe5643e5df6a2a1ede724d31 not found: ID does not exist"
Nov 03 13:20:52.380577 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.380547    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 13:20:52.426496 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:52.426460    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78c58c9bfd-vfmn7"]
Nov 03 13:20:53.682358 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:20:53.682323    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" path="/var/lib/kubelet/pods/b9c74301-9f8c-4263-baf0-2b9b5583ff4c/volumes"
Nov 03 13:20:56.470931 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:20:56.470896    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:21:09.654812 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:09.654769    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:23.506739 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:23.506700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:23.680772 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:23.680739    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:26.471470 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:21:26.471435    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:21:27.947431 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:27.947370    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:28.150601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:28.150564    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:30.615310 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:30.615272    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:32.631895 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:32.631845    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:33.463389 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:33.463334    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:33.628229 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:33.628193    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:34.638199 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:34.638175    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:21:34.693661 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:34.693637    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:21:50.892853 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:21:50.892818    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:21:56.470987 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:21:56.470942    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:22:10.038169 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:10.038137    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:22:17.452302 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452265    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"]
Nov 03 13:22:17.452788 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452772    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2d63a51c-bf03-4100-addc-0264a9058016" containerName="maas-api"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452789    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d63a51c-bf03-4100-addc-0264a9058016" containerName="maas-api"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452801    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" containerName="maas-api"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452807    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" containerName="maas-api"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452822    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" containerName="test-curl-7596-8b12fc"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452827    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" containerName="test-curl-7596-8b12fc"
Nov 03 13:22:17.452837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452836    5784 cpu_manager.go:414] "containerMap: removing container" podUID="708f97d2-541f-4676-a00d-db59163b08bc" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452842    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708f97d2-541f-4676-a00d-db59163b08bc" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452856    5784 cpu_manager.go:414] "containerMap: removing container" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452861    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452869    5784 cpu_manager.go:414] "containerMap: removing container" podUID="284f9899-7554-43e3-bc3d-2dc69e714e14" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452873    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284f9899-7554-43e3-bc3d-2dc69e714e14" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452884    5784 cpu_manager.go:414] "containerMap: removing container" podUID="382baf3a-d92f-4238-813c-01fcd9b0cc69" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452888    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382baf3a-d92f-4238-813c-01fcd9b0cc69" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452896    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452900    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452907    5784 cpu_manager.go:414] "containerMap: removing container" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452911    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452919    5784 cpu_manager.go:414] "containerMap: removing container" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452926    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452938    5784 cpu_manager.go:414] "containerMap: removing container" podUID="afab2659-0852-4586-afb4-770b89cef52a" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452943    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afab2659-0852-4586-afb4-770b89cef52a" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452951    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452955    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452961    5784 cpu_manager.go:414] "containerMap: removing container" podUID="407e4d6e-0b50-4652-b230-13407245d514" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452965    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407e4d6e-0b50-4652-b230-13407245d514" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452971    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452975    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452983    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6413238b-2d42-4ae0-ad5d-5e281f979903" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452987    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6413238b-2d42-4ae0-ad5d-5e281f979903" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.452993    5784 cpu_manager.go:414] "containerMap: removing container" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453003    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453016    5784 cpu_manager.go:414] "containerMap: removing container" podUID="d30341b2-3f0e-4970-ac6c-3f2948df548f" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453021    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30341b2-3f0e-4970-ac6c-3f2948df548f" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453029    5784 cpu_manager.go:414] "containerMap: removing container" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453033    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453040    5784 cpu_manager.go:414] "containerMap: removing container" podUID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453044    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" containerName="maas-api"
Nov 03 13:22:17.453037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453051    5784 cpu_manager.go:414] "containerMap: removing container" podUID="20968277-ad9a-46f5-bc9a-d6384af5fb02" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453055    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20968277-ad9a-46f5-bc9a-d6384af5fb02" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453061    5784 cpu_manager.go:414] "containerMap: removing container" podUID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453065    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453075    5784 cpu_manager.go:414] "containerMap: removing container" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453082    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453091    5784 cpu_manager.go:414] "containerMap: removing container" podUID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" containerName="payload-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453096    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" containerName="payload-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453104    5784 cpu_manager.go:414] "containerMap: removing container" podUID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453108    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453116    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453120    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453188    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="382baf3a-d92f-4238-813c-01fcd9b0cc69" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453198    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453204    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b91d248-e578-4e1c-a954-d281e4c66775" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453211    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="01468df6-1754-4cbe-b664-03e846b35b71" containerName="istio-proxy"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453217    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="afab2659-0852-4586-afb4-770b89cef52a" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453223    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2e6d19-72f9-4327-8bd7-ea7bb992aebf" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453230    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="407e4d6e-0b50-4652-b230-13407245d514" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453239    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6413238b-2d42-4ae0-ad5d-5e281f979903" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453246    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2952de83-4b48-4b89-ba3d-4d2f9deb4e32" containerName="istio-proxy"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453252    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30341b2-3f0e-4970-ac6c-3f2948df548f" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453257    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e0b7e1-c4c1-45db-abd6-669fc5e83a44" containerName="payload-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453262    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e717b5e-fbc6-4aa3-9ec0-33d35e2d1795" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453268    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="20968277-ad9a-46f5-bc9a-d6384af5fb02" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453274    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7fb1c3-dc45-416a-ae66-d14ef00867c9" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453279    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ddced03-6163-4a21-9c27-7b5aa2784db1" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453284    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="1518f5d4-a5dc-4141-a297-e52ad0cf5d80" containerName="payload-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453289    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa2ccfbf-b68c-44a7-8e2c-6384fbe2a565" containerName="payload-pre-processing"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453295    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7919092e-f3e8-491a-987c-303bca4a3486" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453300    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d63a51c-bf03-4100-addc-0264a9058016" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453306    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9415789-a9c5-4ae5-b47f-dc943676cc5f" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453316    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="d018a35a-eb4e-4e16-9c15-4bb3ed19cae8" containerName="test-curl-7596-8b12fc"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453325    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="708f97d2-541f-4676-a00d-db59163b08bc" containerName="maas-api"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453331    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="abc14fde-2904-42ca-b96e-97107c23d314" containerName="main"
Nov 03 13:22:17.454211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.453336    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="284f9899-7554-43e3-bc3d-2dc69e714e14" containerName="main"
Nov 03 13:22:17.456708 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.456687    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.498252 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.498224    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"]
Nov 03 13:22:17.498744 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.498729    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c74301-9f8c-4263-baf0-2b9b5583ff4c" containerName="manager"
Nov 03 13:22:17.501717 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.501703    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.547624 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.547590    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"]
Nov 03 13:22:17.566282 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566245    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98010e5e-33c9-4192-9d80-ea63f8021a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.566485 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566307    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566485 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566450    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566615 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566490    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.566615 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566519    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566615 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566553    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566638    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.566776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566662    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566725    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfddb22-6963-4f01-8912-d98c988b247a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.566776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566743    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.566776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566760    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rfqc\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-kube-api-access-6rfqc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566777    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm9s4\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-kube-api-access-vm9s4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566799    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566864    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566940    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfddb22-6963-4f01-8912-d98c988b247a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566962    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.566987    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.567036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.567013    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.605949 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.605917    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"]
Nov 03 13:22:17.668241 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668210    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfddb22-6963-4f01-8912-d98c988b247a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.668241 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668246    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668262    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rfqc\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-kube-api-access-6rfqc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668279    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm9s4\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-kube-api-access-vm9s4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668298    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668326    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668360    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfddb22-6963-4f01-8912-d98c988b247a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668408    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668446    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668479    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.668516 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668518    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98010e5e-33c9-4192-9d80-ea63f8021a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668579    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668624    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668661    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668688    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668722    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668738    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668767    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668797    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668826    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669033 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.668999    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669636 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.669189    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dfddb22-6963-4f01-8912-d98c988b247a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.669636 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.669443    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98010e5e-33c9-4192-9d80-ea63f8021a5b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.669674    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669754 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.669713    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.669953 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.669931    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.670140 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.670113    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.670317 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.670223    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.671153 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.671124    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.671450 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.671434    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dfddb22-6963-4f01-8912-d98c988b247a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.671645 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.671619    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.671981 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.671964    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dfddb22-6963-4f01-8912-d98c988b247a-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.721457 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.721367    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.721610 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.721532    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.721726 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.721705    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rfqc\" (UniqueName: \"kubernetes.io/projected/4dfddb22-6963-4f01-8912-d98c988b247a-kube-api-access-6rfqc\") pod \"maas-default-gateway-openshift-default-7f5667f86c-cmf42\" (UID: \"4dfddb22-6963-4f01-8912-d98c988b247a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.721857 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.721834    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm9s4\" (UniqueName: \"kubernetes.io/projected/98010e5e-33c9-4192-9d80-ea63f8021a5b-kube-api-access-vm9s4\") pod \"maas-default-gateway-openshift-default-7f5667f86c-sjfwm\" (UID: \"98010e5e-33c9-4192-9d80-ea63f8021a5b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:17.769399 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.769356    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:17.814432 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:17.814403    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:18.183246 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.183215    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"]
Nov 03 13:22:18.184257 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:22:18.184221    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfddb22_6963_4f01_8912_d98c988b247a.slice/crio-0b484615f7c05c510da8cd88d12f200194f9be41afff875137772c6da7cb1810 WatchSource:0}: Error finding container 0b484615f7c05c510da8cd88d12f200194f9be41afff875137772c6da7cb1810: Status 404 returned error can't find the container with id 0b484615f7c05c510da8cd88d12f200194f9be41afff875137772c6da7cb1810
Nov 03 13:22:18.186373 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.186340    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.186468 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.186437    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.186506 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.186472    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.231178 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.231100    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"]
Nov 03 13:22:18.233120 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:22:18.233096    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98010e5e_33c9_4192_9d80_ea63f8021a5b.slice/crio-ac704190bb9744d903ebb6af36951015193015f2bc0b8da53695369e7db4da40 WatchSource:0}: Error finding container ac704190bb9744d903ebb6af36951015193015f2bc0b8da53695369e7db4da40: Status 404 returned error can't find the container with id ac704190bb9744d903ebb6af36951015193015f2bc0b8da53695369e7db4da40
Nov 03 13:22:18.235324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.235294    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.235411 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.235357    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.235461 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.235417    5784 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 13:22:18.711720 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.711679    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" event={"ID":"4dfddb22-6963-4f01-8912-d98c988b247a","Type":"ContainerStarted","Data":"5c3127439a8f589e985617b715ed9c53e7a31944773ed5548cb7abf57811733e"}
Nov 03 13:22:18.712210 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.711740    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" event={"ID":"4dfddb22-6963-4f01-8912-d98c988b247a","Type":"ContainerStarted","Data":"0b484615f7c05c510da8cd88d12f200194f9be41afff875137772c6da7cb1810"}
Nov 03 13:22:18.713260 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.713239    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" event={"ID":"98010e5e-33c9-4192-9d80-ea63f8021a5b","Type":"ContainerStarted","Data":"fc6c95d8c5bbaddff31ae0b2c65a6608db86ea1bfb429d197b7cbf3a323c3f1b"}
Nov 03 13:22:18.713406 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.713282    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" event={"ID":"98010e5e-33c9-4192-9d80-ea63f8021a5b","Type":"ContainerStarted","Data":"ac704190bb9744d903ebb6af36951015193015f2bc0b8da53695369e7db4da40"}
Nov 03 13:22:18.769872 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.769841    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:18.815259 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.815230    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:18.951400 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:18.950970    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podStartSLOduration=1.950948421 podStartE2EDuration="1.950948421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:18.831966116 +0000 UTC m=+5445.274168441" watchObservedRunningTime="2026-11-03 13:22:18.950948421 +0000 UTC m=+5445.393150732"
Nov 03 13:22:19.771021 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:19.770980    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:19.771501 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:19.771037    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:19.852399 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:19.849317    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:19.852399 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:19.849413    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:20.771207 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:20.770739    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:20.771207 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:20.770807    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:20.814939 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:20.814905    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:20.815112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:20.814954    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:21.770400 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:21.769871    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:21.770400 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:21.769922    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:21.815185 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:21.815157    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:21.815662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:21.815202    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:22.769693 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:22.769655    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:22.769964 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:22.769704    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:22.814858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:22.814825    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:22.815025 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:22.814881    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:23.772798 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:23.772749    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:23.773318 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:23.772825    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:23.815118 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:23.815086    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:23.815298 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:23.815136    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:24.770112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:24.770070    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:24.770305 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:24.770141    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:24.815527 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:24.815489    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:24.815936 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:24.815554    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:25.770402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:25.770348    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body=
Nov 03 13:22:25.770616 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:25.770432    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused"
Nov 03 13:22:25.815700 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:25.815661    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 03 13:22:25.816105 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:25.815730    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 03 13:22:26.486246 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:22:26.486203    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:22:27.811556 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:27.811507    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:27.812045 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:27.811590    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:27.816101 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:27.815775    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:27.816101 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:27.815825    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:28.854996 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:28.854484    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:28.854996 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:28.854554    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:28.871401 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:28.867471    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:28.871401 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:28.867545    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.855131 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:29.855094    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.855773 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:29.855738    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:29.868741 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:29.868713    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:29.868948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:29.868912    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:30.900540 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:30.900496    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:30.901058 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:30.900550    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:30.910514 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:30.910484    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:30.910669 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:30.910527    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:31.901822 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.901785    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:31.902320 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.902288    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:31.921515 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.921486    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:31.922908 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.922889    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:31.924051 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.924032    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42"
Nov 03 13:22:31.951561 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.951517    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:22:31.951747 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:31.951722    5784 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:22:32.080206 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:32.080154    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podStartSLOduration=15.080139372 podStartE2EDuration="15.080139372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:18.95569369 +0000 UTC m=+5445.397896018" watchObservedRunningTime="2026-11-03 13:22:32.080139372 +0000 UTC m=+5458.522341734"
Nov 03 13:22:32.236940 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:32.236865    5784 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:32.237176 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:32.237154    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:32.237858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:22:32.237835    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm"
Nov 03 13:22:56.469994 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:22:56.469956    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:23:10.293300 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:23:10.293266    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:26.470884 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:23:26.470798    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:23:31.121548 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:23:31.121485    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:31.719939 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:23:31.719886    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:45.621979 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:23:45.621943    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:23:56.471744 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:23:56.471706    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:24:10.598248 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:10.598210    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:21.303592 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:21.303541    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:24:21.304269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:21.303633    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:26.476732 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:24:26.476699    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:24:30.187835 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.187801    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"]
Nov 03 13:24:30.194734 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.194693    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.243546 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.243506    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:30.243768 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.243739    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:24:30.244173 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.244152    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"]
Nov 03 13:24:30.271858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.271832    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7xb\" (UniqueName: \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.272014 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.271907    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.272014 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.272007    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.373072 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.373037    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.373269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.373108    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.373269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.373152    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7xb\" (UniqueName: \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.373812 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.373783    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.375471 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.375449    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.426327 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.426292    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7xb\" (UniqueName: \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\") pod \"maas-api-revoke-sub-15e7c803-49tk8\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.509133 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.509058    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:30.871250 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:30.871217    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"]
Nov 03 13:24:30.873082 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:24:30.873054    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36177640_0795_45eb_95ed_89607ab48cd9.slice/crio-e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca WatchSource:0}: Error finding container e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca: Status 404 returned error can't find the container with id e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca
Nov 03 13:24:31.663111 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:31.663081    5784 generic.go:356] "Generic (PLEG): container finished" podID="36177640-0795-45eb-95ed-89607ab48cd9" containerID="8c80d245f9cd22e874194ef050668c79fd8f0ba44553ca827b21a0a1312af92e" exitCode=0
Nov 03 13:24:31.663517 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:31.663164    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8" event={"ID":"36177640-0795-45eb-95ed-89607ab48cd9","Type":"ContainerDied","Data":"8c80d245f9cd22e874194ef050668c79fd8f0ba44553ca827b21a0a1312af92e"}
Nov 03 13:24:31.663517 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:31.663223    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8" event={"ID":"36177640-0795-45eb-95ed-89607ab48cd9","Type":"ContainerStarted","Data":"e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca"}
Nov 03 13:24:32.798021 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.797992    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:32.898622 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.898592    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\" (UniqueName: \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\") pod \"36177640-0795-45eb-95ed-89607ab48cd9\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") "
Nov 03 13:24:32.898809 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.898636    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\") pod \"36177640-0795-45eb-95ed-89607ab48cd9\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") "
Nov 03 13:24:32.898809 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.898701    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\") pod \"36177640-0795-45eb-95ed-89607ab48cd9\" (UID: \"36177640-0795-45eb-95ed-89607ab48cd9\") "
Nov 03 13:24:32.899122 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.899101    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca" pod "36177640-0795-45eb-95ed-89607ab48cd9" (UID: "36177640-0795-45eb-95ed-89607ab48cd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:32.900858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.900833    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca" pod "36177640-0795-45eb-95ed-89607ab48cd9" (UID: "36177640-0795-45eb-95ed-89607ab48cd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:32.900975 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:32.900912    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb" pod "36177640-0795-45eb-95ed-89607ab48cd9" (UID: "36177640-0795-45eb-95ed-89607ab48cd9"). InnerVolumeSpecName "kube-api-access-pt7xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:33.000338 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.000255    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7xb\" (UniqueName: \"kubernetes.io/projected/36177640-0795-45eb-95ed-89607ab48cd9-kube-api-access-pt7xb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:33.000338 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.000285    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-xks-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:33.000338 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.000296    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36177640-0795-45eb-95ed-89607ab48cd9-maas-api-openshift-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:33.677967 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.677938    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"
Nov 03 13:24:33.681069 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.681037    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8" event={"ID":"36177640-0795-45eb-95ed-89607ab48cd9","Type":"ContainerDied","Data":"e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca"}
Nov 03 13:24:33.681215 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:33.681081    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1da396bda88061a8e1f0b6c9f921cc0666427ab496d93dbcba4b01c1d66a4ca"
Nov 03 13:24:42.979310 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:42.979220    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:43.197168 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:43.197129    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:43.474009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:43.473977    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:24:48.721562 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.721518    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"]
Nov 03 13:24:48.724076 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.722197    5784 cpu_manager.go:414] "containerMap: removing container" podUID="36177640-0795-45eb-95ed-89607ab48cd9" containerName="revoke-keys"
Nov 03 13:24:48.724076 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.722217    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36177640-0795-45eb-95ed-89607ab48cd9" containerName="revoke-keys"
Nov 03 13:24:48.724076 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.722369    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="36177640-0795-45eb-95ed-89607ab48cd9" containerName="revoke-keys"
Nov 03 13:24:48.725948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.725923    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.787808 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.787777    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"]
Nov 03 13:24:48.819815 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.819783    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 13:24:48.819967 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.819849    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tsgt9\""
Nov 03 13:24:48.857153 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.857109    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.857341 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.857185    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5zb\" (UniqueName: \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.857448 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.857343    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.958437 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.958367    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5zb\" (UniqueName: \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.958662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.958483    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.958662 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.958543    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.959299 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.959271    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:48.961581 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:48.961553    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:49.012445 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:49.011829    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5zb\" (UniqueName: \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\") pod \"maas-api-revoke-sub-f1f310fe-r8b4z\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:49.040223 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:49.040185    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:49.424327 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:49.424121    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"]
Nov 03 13:24:49.798088 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:49.798049    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z" event={"ID":"683203fc-6c95-4d5c-b5dd-b26ed9db2576","Type":"ContainerStarted","Data":"ae10b68be975ec07d984c747de93529035bdc79b58157f257ba62c83dab95b64"}
Nov 03 13:24:50.805452 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:50.805416    5784 generic.go:356] "Generic (PLEG): container finished" podID="683203fc-6c95-4d5c-b5dd-b26ed9db2576" containerID="a48fe4e8f6ce21e9395de0bb88cb296fc28c4529a6c07eb67d6ca8de9e3d6ce5" exitCode=0
Nov 03 13:24:50.805938 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:50.805502    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z" event={"ID":"683203fc-6c95-4d5c-b5dd-b26ed9db2576","Type":"ContainerDied","Data":"a48fe4e8f6ce21e9395de0bb88cb296fc28c4529a6c07eb67d6ca8de9e3d6ce5"}
Nov 03 13:24:51.279991 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:51.277597    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:24:51.279991 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:51.277692    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:24:51.944696 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:51.944667    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:52.013416 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.013361    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\") pod \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") "
Nov 03 13:24:52.013612 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.013426    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\") pod \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") "
Nov 03 13:24:52.013612 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.013464    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\" (UniqueName: \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\") pod \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\" (UID: \"683203fc-6c95-4d5c-b5dd-b26ed9db2576\") "
Nov 03 13:24:52.013752 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.013689    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca" pod "683203fc-6c95-4d5c-b5dd-b26ed9db2576" (UID: "683203fc-6c95-4d5c-b5dd-b26ed9db2576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:24:52.015408 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.015367    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb" pod "683203fc-6c95-4d5c-b5dd-b26ed9db2576" (UID: "683203fc-6c95-4d5c-b5dd-b26ed9db2576"). InnerVolumeSpecName "kube-api-access-vs5zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:24:52.015507 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.015462    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca" pod "683203fc-6c95-4d5c-b5dd-b26ed9db2576" (UID: "683203fc-6c95-4d5c-b5dd-b26ed9db2576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:24:52.114973 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.114886    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-openshift-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:52.114973 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.114917    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/683203fc-6c95-4d5c-b5dd-b26ed9db2576-maas-api-xks-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:52.114973 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.114926    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs5zb\" (UniqueName: \"kubernetes.io/projected/683203fc-6c95-4d5c-b5dd-b26ed9db2576-kube-api-access-vs5zb\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:24:52.818772 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.818747    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"
Nov 03 13:24:52.818926 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.818786    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z" event={"ID":"683203fc-6c95-4d5c-b5dd-b26ed9db2576","Type":"ContainerDied","Data":"ae10b68be975ec07d984c747de93529035bdc79b58157f257ba62c83dab95b64"}
Nov 03 13:24:52.818926 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:24:52.818835    5784 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae10b68be975ec07d984c747de93529035bdc79b58157f257ba62c83dab95b64"
Nov 03 13:24:56.473006 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:24:56.472971    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:25:01.222919 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:01.222881    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:01.823273 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:01.823235    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:02.117346 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:02.117260    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:05.914538 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:05.914501    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.098838 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:06.098803    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.269973 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:06.269881    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:06.305717 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:06.305678    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:25:06.305890 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:06.305783    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:25:09.134832 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:09.134799    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:09.302405 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:09.302343    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:10.935846 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:10.935814    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.178403 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:12.178357    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.349575 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:12.349544    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.548226 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:12.548138    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:12.718472 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:12.718435    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:17.155079 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:17.155046    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:21.987603 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:21.987566    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:26.469953 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:25:26.469921    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:25:26.741273 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:26.741195    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:30.741595 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:30.741567    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:31.355016 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:31.354986    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:31.524552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:31.524515    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:32.072937 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:32.072906    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:34.808710 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:34.808668    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:36.893589 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:36.893553    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:40.967554 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:40.967515    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:41.152114 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:41.152077    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:41.855417 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:41.855358    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:46.534585 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:46.534545    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:51.375786 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:51.375724    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:56.062911 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:25:56.062875    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:25:56.471023 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:25:56.470986    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:26:00.884100 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:00.884058    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:05.663517 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:05.663473    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:10.450435 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:10.450399    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:11.255757 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:11.255674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:15.164258 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:15.164223    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:19.961248 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:19.961212    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:24.788335 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:24.788276    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:26.470794 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:26:26.470754    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:26:29.551325 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:29.551295    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:30.606735 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:30.606700    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:31.009652 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:31.009618    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:34.346174 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:34.346136    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:34.680209 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:34.680182    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:26:34.738712 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:34.738678    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:26:35.565462 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:35.565425    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:36.639559 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:36.639531    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:36.798552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:36.798516    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:37.085958 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:37.085880    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:38.670297 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:38.670265    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:42.986560 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:42.986526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:48.497492 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:48.497448    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:48.672402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:26:48.672338    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:26:56.471257 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:26:56.471219    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:27:11.654034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:11.653995    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:11.818652 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:11.818608    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:12.808789 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:12.808754    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:16.179304 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:16.179270    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:16.762515 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:16.762475    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:18.371149 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:18.371108    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:18.533255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:18.533224    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:26.471729 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:27:26.471690    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:27:32.015519 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:32.015486    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:55.796997 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:27:55.796962    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:27:56.471016 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:27:56.470974    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:28:03.287570 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:28:03.287513    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:28:03.288244 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:28:03.288209    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:28:11.891977 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:28:11.891944    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:26.470656 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:28:26.470612    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:28:37.971229 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:28:37.971197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:47.791393 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:28:47.791352    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:28:56.470434 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:28:56.470389    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:29:09.859488 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:09.859445    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:12.212592 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:12.212513    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:12.375008 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:12.374942    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:26.471096 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:29:26.471058    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:29:33.043276 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:33.043244    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"]
Nov 03 13:29:33.090795 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:33.090765    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e7c803-49tk8"]
Nov 03 13:29:33.680890 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:33.680845    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36177640-0795-45eb-95ed-89607ab48cd9" path="/var/lib/kubelet/pods/36177640-0795-45eb-95ed-89607ab48cd9/volumes"
Nov 03 13:29:43.924233 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:43.924197    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:29:53.043936 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:53.043903    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"]
Nov 03 13:29:53.089588 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:53.089554    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1f310fe-r8b4z"]
Nov 03 13:29:53.681532 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:29:53.681502    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683203fc-6c95-4d5c-b5dd-b26ed9db2576" path="/var/lib/kubelet/pods/683203fc-6c95-4d5c-b5dd-b26ed9db2576/volumes"
Nov 03 13:29:56.470706 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:29:56.470667    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:30:12.594700 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:12.594661    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:19.697714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:19.697674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:20.144404 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:20.144350    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:22.906500 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:22.906461    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:26.508271 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:30:26.508231    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:30:35.480436 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:35.480404    5784 scope.go:122] "RemoveContainer" containerID="8c80d245f9cd22e874194ef050668c79fd8f0ba44553ca827b21a0a1312af92e"
Nov 03 13:30:36.297491 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:36.297454    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:38.217757 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:30:38.217725    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:30:56.469738 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:30:56.469702    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:31:12.890444 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:12.890409    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:26.471151 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:31:26.471107    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:31:27.909926 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:27.909891    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:34.762406 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:34.760649    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:31:34.795834 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:34.795803    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:31:35.504206 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:35.504172    5784 scope.go:122] "RemoveContainer" containerID="a48fe4e8f6ce21e9395de0bb88cb296fc28c4529a6c07eb67d6ca8de9e3d6ce5"
Nov 03 13:31:51.706925 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:31:51.706891    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:31:56.471302 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:31:56.471266    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:32:08.139478 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:32:08.139441    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:08.302480 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:32:08.302447    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:08.716648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:32:08.716607    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:13.382025 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:32:13.381949    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:32:26.473638 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:32:26.473599    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:32:56.481677 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:32:56.481639    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:33:13.616005 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:33:13.615961    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:33:26.472602 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:33:26.472560    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:33:56.473311 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:33:56.473266    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:34:13.958912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:34:13.958871    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:34:26.471675 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:34:26.471632    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:34:56.469940 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:34:56.469905    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:35:14.279658 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:35:14.279578    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:35:26.508982 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:35:26.508944    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:35:56.470611 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:35:56.470568    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:36:14.600111 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:36:14.600071    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:36:26.472489 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:36:26.472446    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:36:34.862997 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:36:34.862961    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:36:34.899701 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:36:34.899661    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:36:56.471716 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:36:56.471676    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:37:14.661185 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:14.661133    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:14.910020 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:14.909982    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:15.384881 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:15.384849    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:24.430530 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:24.430498    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:26.471063 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:37:26.471022    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:37:34.177319 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:34.177280    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:34.818884 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:34.818848    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:35.269852 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:35.269819    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:38.694084 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:38.694051    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:38.874022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:38.873990    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:39.042623 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:39.042536    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:41.922304 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:41.922269    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:42.104328 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:42.104292    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:45.038802 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:45.038765    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:45.407897 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:45.407866    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:50.199993 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:50.199955    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:55.324894 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:37:55.324859    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:37:56.473869 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:37:56.473807    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:38:00.292763 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:00.292674    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:03.500861 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:03.500829    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:04.190490 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:04.190456    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:04.402783 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:04.402751    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:05.687223 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:05.687194    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:06.923753 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:06.923714    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:10.612008 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:10.611973    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:15.235777 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:15.235692    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:15.412222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:15.412191    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:15.588557 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:15.588480    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:20.414913 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:20.414876    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:24.897616 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:24.897550    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:25.051714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:25.051679    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:25.334402 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:25.334307    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:26.471179 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:38:26.471117    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:38:30.147364 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:30.147330    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:35.075887 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:35.075841    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:35.741345 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:35.741314    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:39.844368 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:39.844327    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:44.669255 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:44.669217    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:49.464103 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:49.464017    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:54.231218 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:54.231182    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:38:56.471425 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:38:56.471360    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:38:59.054567 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:38:59.054524    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:03.851009 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:03.850975    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:05.558707 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:05.558673    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:06.018723 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:06.018692    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:08.595337 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:08.595303    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:10.601321 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:10.601289    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:11.919168 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:11.919132    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:13.019036 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:13.019006    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:13.204158 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:13.204104    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:15.622433 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:15.622400    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:17.930283 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:17.930243    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:18.096336 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:18.096306    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:26.473867 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:39:26.473827    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:39:47.468670 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:47.468634    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:47.789632 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:47.789526    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:48.209364 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:48.209329    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:49.933457 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:49.933408    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:50.121316 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:50.121264    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:54.715607 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:54.715575    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:54.907197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:39:54.907159    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:39:56.470711 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:39:56.470610    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:40:00.027171 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:00.027135    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:04.808268 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:04.808181    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:09.689226 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:09.689190    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:14.467614 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:14.467582    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:15.880610 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:15.880576    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:18.708161 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:18.708124    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:20.330194 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:20.330161    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:20.608222 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:20.608110    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:21.250732 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:21.250681    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:21.440408 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:21.440362    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:40:26.475597 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:40:26.475553    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:40:41.730825 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:41.730789    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 13:40:41.731301 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:41.730977    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" podUID="7822d282-6299-48b9-8dc4-df94f622047c" containerName="manager" containerID="cri-o://af1cab7bc8cd374d9ff5b5567e30baad1fe9df860286f44091f811cde0ab764e" gracePeriod=10
Nov 03 13:40:43.016710 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.016663    5784 generic.go:356] "Generic (PLEG): container finished" podID="7822d282-6299-48b9-8dc4-df94f622047c" containerID="af1cab7bc8cd374d9ff5b5567e30baad1fe9df860286f44091f811cde0ab764e" exitCode=0
Nov 03 13:40:43.017193 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.016740    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" event={"ID":"7822d282-6299-48b9-8dc4-df94f622047c","Type":"ContainerDied","Data":"af1cab7bc8cd374d9ff5b5567e30baad1fe9df860286f44091f811cde0ab764e"}
Nov 03 13:40:43.099014 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.098988    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 13:40:43.231628 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.231542    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\" (UniqueName: \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\") pod \"7822d282-6299-48b9-8dc4-df94f622047c\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") "
Nov 03 13:40:43.231628 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.231614    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\") pod \"7822d282-6299-48b9-8dc4-df94f622047c\" (UID: \"7822d282-6299-48b9-8dc4-df94f622047c\") "
Nov 03 13:40:43.232022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.231992    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume" pod "7822d282-6299-48b9-8dc4-df94f622047c" (UID: "7822d282-6299-48b9-8dc4-df94f622047c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 13:40:43.233723 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.233700    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk" pod "7822d282-6299-48b9-8dc4-df94f622047c" (UID: "7822d282-6299-48b9-8dc4-df94f622047c"). InnerVolumeSpecName "kube-api-access-bhrmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:40:43.333396 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.333346    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrmk\" (UniqueName: \"kubernetes.io/projected/7822d282-6299-48b9-8dc4-df94f622047c-kube-api-access-bhrmk\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:43.333396 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:43.333396    5784 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7822d282-6299-48b9-8dc4-df94f622047c-extensions-socket-volume\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:40:44.024112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:44.024032    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w" event={"ID":"7822d282-6299-48b9-8dc4-df94f622047c","Type":"ContainerDied","Data":"b41b8a137bc54936a3972286502de4a9fca6034b35c23cdfd4f56bd94cdd149a"}
Nov 03 13:40:44.024112 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:44.024097    5784 scope.go:122] "RemoveContainer" containerID="af1cab7bc8cd374d9ff5b5567e30baad1fe9df860286f44091f811cde0ab764e"
Nov 03 13:40:44.024587 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:44.024048    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"
Nov 03 13:40:44.224776 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:44.224736    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 13:40:44.270946 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:44.270910    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-db85w"]
Nov 03 13:40:45.680800 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:40:45.680764    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7822d282-6299-48b9-8dc4-df94f622047c" path="/var/lib/kubelet/pods/7822d282-6299-48b9-8dc4-df94f622047c/volumes"
Nov 03 13:40:56.470777 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:40:56.470729    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:41:16.188919 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:16.188832    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:16.341781 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:16.341748    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:41:26.470566 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:41:26.470526    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:41:34.939642 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:34.939610    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:41:34.951335 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:34.951314    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:41:45.442826 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.442789    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"]
Nov 03 13:41:45.443324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443196    5784 cpu_manager.go:414] "containerMap: removing container" podUID="7822d282-6299-48b9-8dc4-df94f622047c" containerName="manager"
Nov 03 13:41:45.443324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443206    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7822d282-6299-48b9-8dc4-df94f622047c" containerName="manager"
Nov 03 13:41:45.443324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443245    5784 cpu_manager.go:414] "containerMap: removing container" podUID="683203fc-6c95-4d5c-b5dd-b26ed9db2576" containerName="revoke-keys"
Nov 03 13:41:45.443324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443250    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683203fc-6c95-4d5c-b5dd-b26ed9db2576" containerName="revoke-keys"
Nov 03 13:41:45.443324 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443313    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="7822d282-6299-48b9-8dc4-df94f622047c" containerName="manager"
Nov 03 13:41:45.443527 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.443329    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="683203fc-6c95-4d5c-b5dd-b26ed9db2576" containerName="revoke-keys"
Nov 03 13:41:45.447648 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.447629    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.492430 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.492396    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"]
Nov 03 13:41:45.535948 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.535919    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lqflp\""
Nov 03 13:41:45.628728 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.628683    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2kc\" (UniqueName: \"kubernetes.io/projected/7795a2cd-150c-4e61-8ef2-b6720472910e-kube-api-access-tk2kc\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.628922 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.628817    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7795a2cd-150c-4e61-8ef2-b6720472910e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.729784 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.729703    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2kc\" (UniqueName: \"kubernetes.io/projected/7795a2cd-150c-4e61-8ef2-b6720472910e-kube-api-access-tk2kc\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.729784 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.729781    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7795a2cd-150c-4e61-8ef2-b6720472910e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.730247 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.730220    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7795a2cd-150c-4e61-8ef2-b6720472910e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:45.794441 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:45.794403    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2kc\" (UniqueName: \"kubernetes.io/projected/7795a2cd-150c-4e61-8ef2-b6720472910e-kube-api-access-tk2kc\") pod \"kuadrant-operator-controller-manager-678d87766f-lxr8c\" (UID: \"7795a2cd-150c-4e61-8ef2-b6720472910e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:46.061167 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:46.061072    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:46.527715 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:46.527690    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"]
Nov 03 13:41:46.529432 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:41:46.529400    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7795a2cd_150c_4e61_8ef2_b6720472910e.slice/crio-a3a493c62dbee8d8d0aed32df7c2a782e82a2eaa3bc5d57cf5e9f3a3ff3b7f5e WatchSource:0}: Error finding container a3a493c62dbee8d8d0aed32df7c2a782e82a2eaa3bc5d57cf5e9f3a3ff3b7f5e: Status 404 returned error can't find the container with id a3a493c62dbee8d8d0aed32df7c2a782e82a2eaa3bc5d57cf5e9f3a3ff3b7f5e
Nov 03 13:41:47.383542 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:47.383501    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c" event={"ID":"7795a2cd-150c-4e61-8ef2-b6720472910e","Type":"ContainerStarted","Data":"095e46351610ddef181f1055d19e0560e0cbe2f3ae414b0b6ba3e95145e1c855"}
Nov 03 13:41:47.383782 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:47.383576    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c" event={"ID":"7795a2cd-150c-4e61-8ef2-b6720472910e","Type":"ContainerStarted","Data":"a3a493c62dbee8d8d0aed32df7c2a782e82a2eaa3bc5d57cf5e9f3a3ff3b7f5e"}
Nov 03 13:41:47.383782 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:47.383605    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:41:47.590507 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:47.590441    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c" podStartSLOduration=2.590427399 podStartE2EDuration="2.590427399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:47.544292128 +0000 UTC m=+6613.986494445" watchObservedRunningTime="2026-11-03 13:41:47.590427399 +0000 UTC m=+6614.032629707"
Nov 03 13:41:56.471773 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:41:56.471731    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:41:58.391917 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:41:58.391888    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-lxr8c"
Nov 03 13:42:16.566258 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:16.566224    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:42:26.471445 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:42:26.471404    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:42:45.113087 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.113006    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:45.130836 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.130804    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.199837 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.199802    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.200022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.199869    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9jrz\" (UniqueName: \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.200022 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.199985    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.200116 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.200044    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.225574 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.225540    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 13:42:45.225770 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.225635    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 13:42:45.225770 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.225688    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9vv49\""
Nov 03 13:42:45.229690 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.229664    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:45.301520 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.301485    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9jrz\" (UniqueName: \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.301732 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.301562    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.301732 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.301623    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.301732 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.301663    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.302514 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.302484    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.304243 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.304222    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.304341 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.304287    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.355411 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.355369    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9jrz\" (UniqueName: \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\") pod \"maas-api-6f554dd4c5-84kbk\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") " pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.445279 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.445245    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:45.807446 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.807421    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:45.809176 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:42:45.809152    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02242d6_e483_4de5_b1a5_b604dc79288d.slice/crio-69a7625270422a36d2f4d48cc49f91ec6ee7126f08fd7909edb8d96715849652 WatchSource:0}: Error finding container 69a7625270422a36d2f4d48cc49f91ec6ee7126f08fd7909edb8d96715849652: Status 404 returned error can't find the container with id 69a7625270422a36d2f4d48cc49f91ec6ee7126f08fd7909edb8d96715849652
Nov 03 13:42:45.810569 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:45.810551    5784 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 13:42:46.789752 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:46.789718    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" event={"ID":"b02242d6-e483-4de5-b1a5-b604dc79288d","Type":"ContainerStarted","Data":"69a7625270422a36d2f4d48cc49f91ec6ee7126f08fd7909edb8d96715849652"}
Nov 03 13:42:47.797767 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:47.797719    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" event={"ID":"b02242d6-e483-4de5-b1a5-b604dc79288d","Type":"ContainerStarted","Data":"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"}
Nov 03 13:42:47.798183 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:47.797823    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:47.904915 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:47.904856    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" podStartSLOduration=1.94925922 podStartE2EDuration="2.904843242s" totalImagesPullingTime="955.584022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:47.904210428 +0000 UTC m=+6674.346412733" watchObservedRunningTime="2026-11-03 13:42:47.904843242 +0000 UTC m=+6674.347045548"
Nov 03 13:42:53.809858 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.809830    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:53.827004 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.826972    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:53.841567 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.841510    5784 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" podUID="b02242d6-e483-4de5-b1a5-b604dc79288d" containerName="maas-api" containerID="cri-o://b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0" gracePeriod=30
Nov 03 13:42:53.873143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.873111    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"]
Nov 03 13:42:53.876983 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.876966    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:53.963581 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.963552    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"]
Nov 03 13:42:53.988105 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.988065    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgvz7\" (UniqueName: \"kubernetes.io/projected/d932f395-0e65-4ee6-89a7-90b94a60ef14-kube-api-access-dgvz7\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:53.988290 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.988127    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-metrics-certs\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:53.988290 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.988230    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-maas-api-tls\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:53.988403 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:53.988367    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d932f395-0e65-4ee6-89a7-90b94a60ef14-openshift-service-ca\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.089649 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.089616    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgvz7\" (UniqueName: \"kubernetes.io/projected/d932f395-0e65-4ee6-89a7-90b94a60ef14-kube-api-access-dgvz7\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.089816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.089695    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-metrics-certs\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.089816 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.089731    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-maas-api-tls\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.089939 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.089814    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d932f395-0e65-4ee6-89a7-90b94a60ef14-openshift-service-ca\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.090527 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.090501    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d932f395-0e65-4ee6-89a7-90b94a60ef14-openshift-service-ca\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.092063 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.092044    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-metrics-certs\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.092146 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.092074    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d932f395-0e65-4ee6-89a7-90b94a60ef14-maas-api-tls\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.108338 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.108315    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:54.143545 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.143517    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgvz7\" (UniqueName: \"kubernetes.io/projected/d932f395-0e65-4ee6-89a7-90b94a60ef14-kube-api-access-dgvz7\") pod \"maas-api-6c989b4c99-fx6jz\" (UID: \"d932f395-0e65-4ee6-89a7-90b94a60ef14\") " pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.190934 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.190840    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\") pod \"b02242d6-e483-4de5-b1a5-b604dc79288d\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") "
Nov 03 13:42:54.191098 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.190997    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\") pod \"b02242d6-e483-4de5-b1a5-b604dc79288d\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") "
Nov 03 13:42:54.191098 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.191055    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\") pod \"b02242d6-e483-4de5-b1a5-b604dc79288d\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") "
Nov 03 13:42:54.191211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.191194    5784 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\" (UniqueName: \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\") pod \"b02242d6-e483-4de5-b1a5-b604dc79288d\" (UID: \"b02242d6-e483-4de5-b1a5-b604dc79288d\") "
Nov 03 13:42:54.191428 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.191366    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca" pod "b02242d6-e483-4de5-b1a5-b604dc79288d" (UID: "b02242d6-e483-4de5-b1a5-b604dc79288d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 13:42:54.191598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.191580    5784 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b02242d6-e483-4de5-b1a5-b604dc79288d-openshift-service-ca\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:54.192952 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.192927    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls" pod "b02242d6-e483-4de5-b1a5-b604dc79288d" (UID: "b02242d6-e483-4de5-b1a5-b604dc79288d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:54.193107 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.193092    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs" pod "b02242d6-e483-4de5-b1a5-b604dc79288d" (UID: "b02242d6-e483-4de5-b1a5-b604dc79288d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 13:42:54.193161 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.193146    5784 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz" pod "b02242d6-e483-4de5-b1a5-b604dc79288d" (UID: "b02242d6-e483-4de5-b1a5-b604dc79288d"). InnerVolumeSpecName "kube-api-access-j9jrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 13:42:54.216868 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.216841    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:54.292601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.292567    5784 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9jrz\" (UniqueName: \"kubernetes.io/projected/b02242d6-e483-4de5-b1a5-b604dc79288d-kube-api-access-j9jrz\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:54.292601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.292597    5784 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-maas-api-tls\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:54.292601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.292608    5784 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b02242d6-e483-4de5-b1a5-b604dc79288d-metrics-certs\") on node \"ip-10-0-139-140.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 13:42:54.585547 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.585522    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"]
Nov 03 13:42:54.587395 ip-10-0-139-140 kubenswrapper[5784]: W1103 13:42:54.587352    5784 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd932f395_0e65_4ee6_89a7_90b94a60ef14.slice/crio-aa079b3605a808f1b1858e1759bcd5d0cb8eefa09e3da67cc145f331e8ebf889 WatchSource:0}: Error finding container aa079b3605a808f1b1858e1759bcd5d0cb8eefa09e3da67cc145f331e8ebf889: Status 404 returned error can't find the container with id aa079b3605a808f1b1858e1759bcd5d0cb8eefa09e3da67cc145f331e8ebf889
Nov 03 13:42:54.848555 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.848474    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz" event={"ID":"d932f395-0e65-4ee6-89a7-90b94a60ef14","Type":"ContainerStarted","Data":"aa079b3605a808f1b1858e1759bcd5d0cb8eefa09e3da67cc145f331e8ebf889"}
Nov 03 13:42:54.849738 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.849714    5784 generic.go:356] "Generic (PLEG): container finished" podID="b02242d6-e483-4de5-b1a5-b604dc79288d" containerID="b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0" exitCode=0
Nov 03 13:42:54.849850 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.849756    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" event={"ID":"b02242d6-e483-4de5-b1a5-b604dc79288d","Type":"ContainerDied","Data":"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"}
Nov 03 13:42:54.849850 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.849765    5784 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"
Nov 03 13:42:54.849850 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.849784    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk" event={"ID":"b02242d6-e483-4de5-b1a5-b604dc79288d","Type":"ContainerDied","Data":"69a7625270422a36d2f4d48cc49f91ec6ee7126f08fd7909edb8d96715849652"}
Nov 03 13:42:54.849850 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.849804    5784 scope.go:122] "RemoveContainer" containerID="b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"
Nov 03 13:42:54.866700 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.866677    5784 scope.go:122] "RemoveContainer" containerID="b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"
Nov 03 13:42:54.866988 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:42:54.866969    5784 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0\": container with ID starting with b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0 not found: ID does not exist" containerID="b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"
Nov 03 13:42:54.867045 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:54.866998    5784 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0"} err="failed to get container status \"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0\": rpc error: code = NotFound desc = could not find container \"b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0\": container with ID starting with b060a96f66bb46d004b1598907cbf975cd58e4a4f8112eef565c284c743bfdc0 not found: ID does not exist"
Nov 03 13:42:55.001211 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.001177    5784 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:55.047286 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.047252    5784 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f554dd4c5-84kbk"]
Nov 03 13:42:55.680473 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.680444    5784 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b02242d6-e483-4de5-b1a5-b604dc79288d" path="/var/lib/kubelet/pods/b02242d6-e483-4de5-b1a5-b604dc79288d/volumes"
Nov 03 13:42:55.858908 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.858817    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz" event={"ID":"d932f395-0e65-4ee6-89a7-90b94a60ef14","Type":"ContainerStarted","Data":"5ddc576b3a5551aad7ee6b4a8a4c8233262696e1eba65318a7bb8d9dae0db2e9"}
Nov 03 13:42:55.859367 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.858943    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:42:55.968392 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:42:55.968321    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz" podStartSLOduration=2.029368644 podStartE2EDuration="2.968306854s" totalImagesPullingTime="938.93821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:55.967017223 +0000 UTC m=+6682.409219657" watchObservedRunningTime="2026-11-03 13:42:55.968306854 +0000 UTC m=+6682.410509158"
Nov 03 13:42:56.471568 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:42:56.471532    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:43:01.870633 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:43:01.870604    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6c989b4c99-fx6jz"
Nov 03 13:43:04.585011 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:43:04.584959    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:06.312605 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:43:06.312562    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-lxmnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:43:06.313034 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:43:06.312632    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-lxmnf" podUID="90ec043c-9364-4fa0-a313-a236c85aa364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:43:16.815152 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:43:16.815113    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:43:26.474706 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:43:26.474664    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:43:56.470645 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:43:56.470607    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:44:18.485389 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:44:18.485292    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:18.645739 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:44:18.645694    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:26.471676 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:44:26.471639    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:44:39.238571 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:44:39.238536    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:52.931716 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:44:52.931683    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:44:56.470816 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:44:56.470716    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:45:02.975603 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:02.975516    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:02.976647 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:02.976605    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:03.275850 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:03.275741    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-sjfwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:03.276037 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:03.275836    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-sjfwm" podUID="98010e5e-33c9-4192-9d80-ea63f8021a5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:17.704558 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:17.704521    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:45:17.965062 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:17.964966    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:45:17.965062 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:45:17.965029    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:45:26.475151 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:45:26.475104    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:45:56.471092 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:45:56.471052    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:46:18.031510 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:46:18.031478    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:46:26.470765 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:46:26.470718    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:46:34.984337 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:46:34.984306    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:46:34.996998 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:46:34.996975    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmxdcz_7ef52971-6bf7-43b3-890e-f2752b770cb2/extract/0.log"
Nov 03 13:46:56.471649 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:46:56.471611    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:47:18.387602 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:47:18.387517    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:47:26.470954 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:47:26.470914    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:47:56.478803 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:47:56.478761    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:48:17.973550 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:48:17.973505    5784 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-cmf42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 13:48:17.974340 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:48:17.974304    5784 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-cmf42" podUID="4dfddb22-6963-4f01-8912-d98c988b247a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 13:48:18.623005 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:48:18.622974    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:19.568248 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:48:19.568213    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:19.759494 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:48:19.759458    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:48:26.478286 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:48:26.478243    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:48:56.472957 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:48:56.472912    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:49:02.370915 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:02.370883    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fccf55576-2gn95_e9117e63-a8c2-4c65-a7d0-adead6725e93/authorino/0.log"
Nov 03 13:49:14.942203 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:14.942171    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-np6xb_19d9ca6d-3426-4d37-86e9-f7051fc5ce41/manager/0.log"
Nov 03 13:49:15.214434 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:15.214349    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cc94k_ce713df0-9f1c-4581-8971-6dbc92a58b67/manager/0.log"
Nov 03 13:49:15.516552 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:15.516475    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rxmxg_4e96077b-bec2-45f0-97bc-58f8f620f925/manager/0.log"
Nov 03 13:49:17.213401 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:17.213346    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q6gzj_4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4/proxy/0.log"
Nov 03 13:49:18.471510 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:18.471476    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6c989b4c99-fx6jz_d932f395-0e65-4ee6-89a7-90b94a60ef14/maas-api/0.log"
Nov 03 13:49:18.987241 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:18.987206    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:22.206293 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:22.206255    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2xh4z_0a15fffd-279c-47b8-9a54-2e60a6ec367a/postgres/0.log"
Nov 03 13:49:24.150795 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:24.150763    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fccf55576-2gn95_e9117e63-a8c2-4c65-a7d0-adead6725e93/authorino/0.log"
Nov 03 13:49:24.505537 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:24.505457    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m9n58_f248db66-6a89-4874-92c0-d448e2380828/manager/0.log"
Nov 03 13:49:24.743042 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:24.743014    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kpq2r_7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8/manager/0.log"
Nov 03 13:49:25.202464 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:25.202432    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lxr8c_7795a2cd-150c-4e61-8ef2-b6720472910e/manager/0.log"
Nov 03 13:49:25.856726 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:25.856685    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pw5vh_f993bb43-355f-439b-9add-9ec55f05ed03/manager/0.log"
Nov 03 13:49:26.470829 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:49:26.470793    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:49:26.856847 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:26.856771    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x_a1f5b1de-4ae6-44aa-b7e4-00b3120310e5/istio-proxy/0.log"
Nov 03 13:49:27.946468 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:27.946437    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cmf42_4dfddb22-6963-4f01-8912-d98c988b247a/istio-proxy/0.log"
Nov 03 13:49:28.699601 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:28.699568    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lxmnf_90ec043c-9364-4fa0-a313-a236c85aa364/istio-proxy/0.log"
Nov 03 13:49:28.926779 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:28.926743    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-sjfwm_98010e5e-33c9-4192-9d80-ea63f8021a5b/istio-proxy/0.log"
Nov 03 13:49:30.130881 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:30.130852    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6_627077e6-5541-4411-b611-a41dcd52ed1b/istio-proxy/0.log"
Nov 03 13:49:43.792605 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:43.792573    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:44.009583 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:44.009548    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:49:53.083598 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:53.083564    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8tx9f_3ece716d-9af9-434f-85bf-9e6e6fb988f3/global-pull-secret-syncer/0.log"
Nov 03 13:49:56.144130 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:56.144100    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log"
Nov 03 13:49:56.471053 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:49:56.470905    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:49:58.293643 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:49:58.293612    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-140.us-west-2.compute.internal_84ff2cd276a94f8c36394a4ab321034f/haproxy/0.log"
Nov 03 13:50:03.236530 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:03.236495    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fccf55576-2gn95_e9117e63-a8c2-4c65-a7d0-adead6725e93/authorino/0.log"
Nov 03 13:50:04.463288 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.463259    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-m9n58_f248db66-6a89-4874-92c0-d448e2380828/manager/0.log"
Nov 03 13:50:04.621541 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.621506    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-np6xb_19d9ca6d-3426-4d37-86e9-f7051fc5ce41/manager/0.log"
Nov 03 13:50:04.721760 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.721682    5784 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"]
Nov 03 13:50:04.722143 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.722130    5784 cpu_manager.go:414] "containerMap: removing container" podUID="b02242d6-e483-4de5-b1a5-b604dc79288d" containerName="maas-api"
Nov 03 13:50:04.722197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.722144    5784 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b02242d6-e483-4de5-b1a5-b604dc79288d" containerName="maas-api"
Nov 03 13:50:04.722234 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.722222    5784 memory_manager.go:365] "RemoveStaleState removing state" podUID="b02242d6-e483-4de5-b1a5-b604dc79288d" containerName="maas-api"
Nov 03 13:50:04.725584 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.725563    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.774191 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.774162    5784 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7fst8\"/\"default-dockercfg-mjtw8\""
Nov 03 13:50:04.785264 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.785238    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-proc\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.785418 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.785281    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-sys\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.785418 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.785362    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-lib-modules\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.785502 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.785417    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbmjd\" (UniqueName: \"kubernetes.io/projected/d9555f90-e03c-433e-acb1-208a55665ef0-kube-api-access-xbmjd\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.785538 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.785511    5784 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-podres\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.823156 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.823126    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"]
Nov 03 13:50:04.866922 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.866883    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"kube-root-ca.crt\""
Nov 03 13:50:04.867189 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.867161    5784 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7fst8\"/\"openshift-service-ca.crt\""
Nov 03 13:50:04.874030 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.874008    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-kpq2r_7d485060-1b5d-41c9-8b5c-ba2d4da0bfa8/manager/0.log"
Nov 03 13:50:04.886231 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886210    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-sys\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886340 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886257    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-lib-modules\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886398 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886329    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-sys\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886398 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886369    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbmjd\" (UniqueName: \"kubernetes.io/projected/d9555f90-e03c-433e-acb1-208a55665ef0-kube-api-access-xbmjd\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886438    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-lib-modules\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886474 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886442    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-podres\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886579 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886535    5784 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-proc\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886639 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886612    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-proc\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.886686 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.886565    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9555f90-e03c-433e-acb1-208a55665ef0-podres\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:04.932243 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.932222    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cc94k_ce713df0-9f1c-4581-8971-6dbc92a58b67/manager/0.log"
Nov 03 13:50:04.940417 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:04.940390    5784 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbmjd\" (UniqueName: \"kubernetes.io/projected/d9555f90-e03c-433e-acb1-208a55665ef0-kube-api-access-xbmjd\") pod \"perf-node-gather-daemonset-6qsk2\" (UID: \"d9555f90-e03c-433e-acb1-208a55665ef0\") " pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:05.037917 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.037827    5784 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:05.422789 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.422753    5784 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"]
Nov 03 13:50:05.521967 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.521938    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rxmxg_4e96077b-bec2-45f0-97bc-58f8f620f925/manager/0.log"
Nov 03 13:50:05.729793 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.729697    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-lxr8c_7795a2cd-150c-4e61-8ef2-b6720472910e/manager/0.log"
Nov 03 13:50:05.865527 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.865490    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2" event={"ID":"d9555f90-e03c-433e-acb1-208a55665ef0","Type":"ContainerStarted","Data":"d9e88ddc76d598d4a315fad81b5f943a05c815415ef169b0b9841dee06a285df"}
Nov 03 13:50:05.865714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.865545    5784 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2" event={"ID":"d9555f90-e03c-433e-acb1-208a55665ef0","Type":"ContainerStarted","Data":"b3694609fd997fb8f1ee896ab47988c07de17c93f1148781b5c6d14b2867085c"}
Nov 03 13:50:05.865714 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.865590    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:05.973517 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:05.973465    5784 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2" podStartSLOduration=1.9734502269999998 podStartE2EDuration="1.973450227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:05.971309398 +0000 UTC m=+7112.413511704" watchObservedRunningTime="2026-11-03 13:50:05.973450227 +0000 UTC m=+7112.415652581"
Nov 03 13:50:06.585026 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:06.584991    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-pw5vh_f993bb43-355f-439b-9add-9ec55f05ed03/manager/0.log"
Nov 03 13:50:07.119556 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:07.119523    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q6gzj_4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4/proxy/0.log"
Nov 03 13:50:09.246127 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:09.246090    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/alertmanager/0.log"
Nov 03 13:50:09.653579 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:09.653551    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/config-reloader/0.log"
Nov 03 13:50:10.058482 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:10.058403    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/kube-rbac-proxy-web/0.log"
Nov 03 13:50:10.272912 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:10.272885    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-np6xb_19d9ca6d-3426-4d37-86e9-f7051fc5ce41/manager/0.log"
Nov 03 13:50:10.464289 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:10.464266    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/kube-rbac-proxy/0.log"
Nov 03 13:50:10.533269 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:10.533239    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cc94k_ce713df0-9f1c-4581-8971-6dbc92a58b67/manager/0.log"
Nov 03 13:50:10.878869 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:10.878849    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/kube-rbac-proxy-metric/0.log"
Nov 03 13:50:11.248514 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:11.248402    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rxmxg_4e96077b-bec2-45f0-97bc-58f8f620f925/manager/0.log"
Nov 03 13:50:11.293188 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:11.293151    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/prom-label-proxy/0.log"
Nov 03 13:50:11.698549 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:11.698517    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b299cb60-8c1f-4a01-befa-fece4d244bbb/init-config-reloader/0.log"
Nov 03 13:50:11.882142 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:11.882115    5784 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7fst8/perf-node-gather-daemonset-6qsk2"
Nov 03 13:50:12.529230 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:12.529201    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fmr2s_6c439b40-37df-4d6b-82ce-004d3275ad8e/kube-state-metrics/0.log"
Nov 03 13:50:12.936020 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:12.935991    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fmr2s_6c439b40-37df-4d6b-82ce-004d3275ad8e/kube-rbac-proxy-main/0.log"
Nov 03 13:50:13.035048 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:13.035018    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q6gzj_4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4/proxy/0.log"
Nov 03 13:50:13.341241 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:13.341140    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fmr2s_6c439b40-37df-4d6b-82ce-004d3275ad8e/kube-rbac-proxy-self/0.log"
Nov 03 13:50:13.751804 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:13.751776    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-96685b96b-5s9hm_49f19698-00a5-4e9e-a7c3-3fd1a8742421/metrics-server/0.log"
Nov 03 13:50:16.449120 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:16.449093    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-np6xb_19d9ca6d-3426-4d37-86e9-f7051fc5ce41/manager/0.log"
Nov 03 13:50:16.724952 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:16.724863    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-cc94k_ce713df0-9f1c-4581-8971-6dbc92a58b67/manager/0.log"
Nov 03 13:50:17.109355 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:17.109322    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrh8c_a36293e5-2420-4631-ad8b-5747f97896f1/node-exporter/0.log"
Nov 03 13:50:17.374803 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:17.374723    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-rxmxg_4e96077b-bec2-45f0-97bc-58f8f620f925/manager/0.log"
Nov 03 13:50:17.517930 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:17.517894    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrh8c_a36293e5-2420-4631-ad8b-5747f97896f1/kube-rbac-proxy/0.log"
Nov 03 13:50:17.922351 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:17.922323    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wrh8c_a36293e5-2420-4631-ad8b-5747f97896f1/init-textfile/0.log"
Nov 03 13:50:18.330727 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:18.330650    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pmqqh_5e91a382-0182-42f7-aa04-29c13b922a49/kube-rbac-proxy-main/0.log"
Nov 03 13:50:18.735579 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:18.735545    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pmqqh_5e91a382-0182-42f7-aa04-29c13b922a49/kube-rbac-proxy-self/0.log"
Nov 03 13:50:19.093533 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:19.093455    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-q6gzj_4418bc2e-326b-4c2d-b351-5f9c7fbe1ab4/proxy/0.log"
Nov 03 13:50:19.140200 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:19.140170    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pmqqh_5e91a382-0182-42f7-aa04-29c13b922a49/openshift-state-metrics/0.log"
Nov 03 13:50:19.302153 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:19.302120    5784 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-785td_00433242-908e-469a-8d10-39330e427e5a/konnectivity-agent/0.log" limit=1024
Nov 03 13:50:22.415243 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:22.415213    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dm9h8_7aaede73-1b46-4d67-a336-377d44b8c286/prometheus-operator/0.log"
Nov 03 13:50:22.591048 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:22.591014    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f79bp2x_a1f5b1de-4ae6-44aa-b7e4-00b3120310e5/istio-proxy/0.log"
Nov 03 13:50:22.814585 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:22.814513    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-dm9h8_7aaede73-1b46-4d67-a336-377d44b8c286/kube-rbac-proxy/0.log"
Nov 03 13:50:23.635197 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:23.635168    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5895657d88-9n7mx_b77b9219-acad-4786-91ad-6bfe81c64ffc/telemeter-client/0.log"
Nov 03 13:50:24.048766 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:24.048647    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5895657d88-9n7mx_b77b9219-acad-4786-91ad-6bfe81c64ffc/reload/0.log"
Nov 03 13:50:24.453229 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:24.453169    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5895657d88-9n7mx_b77b9219-acad-4786-91ad-6bfe81c64ffc/kube-rbac-proxy/0.log"
Nov 03 13:50:24.869396 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:24.869280    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/thanos-query/0.log"
Nov 03 13:50:25.271892 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:25.271846    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/kube-rbac-proxy-web/0.log"
Nov 03 13:50:25.589138 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:25.589059    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-cmf42_4dfddb22-6963-4f01-8912-d98c988b247a/istio-proxy/0.log"
Nov 03 13:50:25.674293 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:25.674261    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/kube-rbac-proxy/0.log"
Nov 03 13:50:26.078245 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:26.078218    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/prom-label-proxy/0.log"
Nov 03 13:50:26.242148 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:26.242121    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-lxmnf_90ec043c-9364-4fa0-a313-a236c85aa364/istio-proxy/0.log"
Nov 03 13:50:26.462557 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:26.462526    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-sjfwm_98010e5e-33c9-4192-9d80-ea63f8021a5b/istio-proxy/0.log"
Nov 03 13:50:26.471921 ip-10-0-139-140 kubenswrapper[5784]: E1103 13:50:26.471888    5784 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-gk9d8" containerName="extractor"
Nov 03 13:50:26.484041 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:26.484014    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/kube-rbac-proxy-rules/0.log"
Nov 03 13:50:26.886463 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:26.886435    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7748596f65-48dhl_f14e9e71-6399-459d-9ade-004b2390424a/kube-rbac-proxy-metrics/0.log"
Nov 03 13:50:27.550565 ip-10-0-139-140 kubenswrapper[5784]: I1103 13:50:27.550487    5784 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-65dccbd94c-zf8n6_627077e6-5541-4411-b611-a41dcd52ed1b/istio-proxy/0.log"